That’s a good example, I believe it could be similar case like with RAG. I didn’t hear for a while about any recommendations to use RAG.

I don’t think subagents are the problem. Uncontrolled fan-out is. Separate contexts and independent analysis are extremely useful for larger tasks. In my experience, Pro is far too limited for sustained programming, although that probably depends a lot on the kind of engineering work you do.

Fable is great, but overkill for most tasks. Sonnet and Opus are already good enough for the execution work. I find Fable much more useful for planning, orchestration, advising, and judgment.

I am not sure. I think of myself as a pro user yet I am finding hard time maxing out Pro tier. It only happens with sub-agents without any material returns.

The audit point is especially interesting. Is your Claude token-audit tool available anywhere?

I didn't update it for a while. It reads your local conversation history, correlate input/output token with usage (claude doesn't give exact breakdown), and classify if they are subagent/tooluse/normal message/whatever. Let me know if I can make it any way useful to you.

https://github.com/kmcheung12/ccaudit

Thanks, this looks useful. The subagent and tool-use breakdown is exactly the kind of visibility I was looking for. I will give it a try.

Very similar experience. The review UI might actually be the most important part of the whole pipeline. Does the confidence ranking match what you end up rejecting?

It's a future evaluation for me, as I'm keeping the app with a "Data Not Collected" App Store label for now. Once I have some visibility to stats of how people answer each question and at which confidence level I expect it to reveal bad/poorly worded questions that made it through my pipeline.

Also on my to do list is an upgrade similar to my de-duplication logic. Currently I pass in my approved set of questions as a list of topics and questions to avoid. It works well especially when combined with the more deterministic comparison of generated questions vs approved question by the separate model. The upgrade would be to pass in the questions REJECTED by me. These are question which made it through ever automated gauntlet of models and code, yet still ultimately were sent to the waste bin by human review.

I'm imagining that by including the list of examples of what NOT to do, it could inform the initial Generation to be smarter. I currently have a limited list of a hand picked "bad question shapes" where I provide the question and why its bad kept statically in my prompt. But I've realized sorting through thousands of questions, I've probably built implicit data through rejection that I may not actually recognize myself which the model may pick up on.

At the shell level, the agent does not need to know about Bash. The contents of "$OUT" are passed to echo as one argument, then sent to claude-mem-save over stdin. Bash does not re-evaluate them as shell code. printf '%s\n' "$OUT" would be more robust than echo. Whether claude-mem-save can use arbitrary output is a separate question.

there are a million ways both of those can break.

but the other comment point this out is already dead and buried. lol. so much cope from vibecoders.

That can work, especially for privacy or repetitive tasks. My experiment focused on subscriptions I already had, but local models are a natural addition to the routing layer.

Thanks for sharing. That’s a very interesting direction. I will take a look.

Good point, thanks. Of course I was not able to fully eliminate hallucinations, but significantly reduce

looking promising, will give it a try! thanks for sharing

Try to mix both subscribtions and take the best from both providers. In my opinion, we should not just one over another but understand the power of both and mix them to reach the better results.