Important to note is that what is being measured here is the ability of the models not of the chat tools themselves, which combine model completions with other tools that the models can call upon. The mainstream labs already know this about models, it's no secret, and in fact training materials from e.g. Anthropic are at pains to point out that users, or analysts designing workflows, have the reponsibility to ensure the correct tools are used and that human verification takes place at appropriate stages depending on the risk/consequences of the task at hand.
Of course a language-completion model with a training cutoff date won't have up-to-date information on tax rules or the ability to carry out correct numerical calculations, but when you combine that with (in Claude terminology) web search and code execution tools invoked by the chat agent, you immediately have much more reliable results.
I keep finding that the current harnesses, when encountering syntax that was invalid at training time but is now valid due to new language versions or custom extensions, don't correctly figure out why and assume something is wrong with the codebase or toolchain. I would hate to have that happen with my taxes.
You do you. Just carry on blacksmithing in your forge. People are always going to need swords right? Any seismic changes to society that change that fact can only be the result of evil, selfish people which surely someone else will put a stop to before you find yourself out of a job. Meanwhile all the other people called Smith also have families to feed...
> I don't care if it's "intelligent", I don't care if it "has a mind"...
Agreed. The AI is useful for the particular tasks that it proves itself useful for. And if it occasionally spits out a claim that it is "genuinely curious" about a piece of research that I will have to do myself because it turned out to be beyond its mechanical capabilities, then it's more productive for me to simply ignore that claim as a statistical anomaly - a mere hallucination - rather than allowing it to burn a ton of extra tokens outputting what may or may not be the current state of the art on theory-of-mind applied to LLMs because I make the mistake of telling it that it might not actually be capable of experiencing emotion.
ARPAnet definitely wasn't built "by" the military.
"For" the military is close enough, although "defense research" would probably be more accurate than "military" given that initial users were centered around large universities.
From the start the DoD had issues with the free and open culture from the userbase that skewed towards academic types, later splitting off into a restricted MILNET after a few years.
Models are already good enough. Last week I had Fable plan out a project that took approx 4 days end to end with each phase orchestrated by a supervisor agent delegating individual tasks to other agents, coordinating everything and checking status by simple text files in the repo. I didn’t have to tell the agent to do it that way, it just came up with it and set up the infra as part of the planning overview. Great that everyone’s posting their “my secret sauce” cookbooks just to jump on the hype train but it looks like the models have already figured it out for themselves.
What is the interplay between such delegation and token cache timeout? If Fable is truly active for 4 days, enough to keep the cache hot, the cost would be astronomical. OTOH, if Fable idles while subagents are active, then each awakening is a cache miss.
True, that’s why I didn’t have a single Fable thread running for the whole time. Each phase had its own orchestrator that ran about 4-5h, in Fable or Opus depending on complexity, delegating to sub agents along the way - they had tasks broken down in a granular enough way that each returned in anything from 5 to 30 mins approx, then the lead agent did some validating and status updates before delegating the next step. So the main agent’s cache never went cold. Each phase raised its own PR and had an opportunity for human validation/code review before the next phase kicked off in a new context.
> Great that everyone’s posting their “my secret sauce” cookbooks just to jump on the hype train but it looks like the models have already figured it out for themselves
Unfortunate lesson to be learned here: there's not much leverage here other than just using AI. Previously, us devs could get a head start and build some institutional knowledge but not this time. I'm bearish on all the custom harnesses stuff that people talk about.
What helps me is to understand the failure modes of LLMs - it can't be articulated in easy words but something you can learn slightly by just using it. For example I have an intuition of when to start compacting but Codex already does it for you now haha.
My take: the highest leverage move for us is to write AGENTS.md and provide everything that the model can't learn on its own or might take time to learn.
One piece of anecdata: after first getting into Claude Code and then discovered the Claude app I briefly got excited about using it in voice chat mode, demonstrated this to my Austrian wife who found it immediately cool then of course asked if she could chat with it in German. On finding the answer was no, she immediately signed up for ChatGPT so she could use the same feature in her native language. I’m sure Anthropic models are decent with German in text chat, but the lack of that one feature is what drove her straight to a competitor.
I mean, technically it was built for "the management of general information about accelerators and experiments at CERN". The fact that it also supports a whole bunch of peoples' passion projects as well as a multi-trillion dollar commercial marketplace is just a bonus...
Sorry you're struggling. There are tons of resources out there from people who've been through the same pain and built up techniques to mitigate them. Matt Pocock's YouTube channel is one good starting place, there are many others. One key tip though is - you own the architecture of your application. If your files have become bloated over several rounds of LLM-generated code, that is primarily your responsibility to observe and push back on that, to ensure your repo as a brief but firm AGENTS.md or CLAUDE.md to describe architectural non-negotiables and some kind of /code-review skill that you give to a second agent to review the first agent's work against those standards. Worried about LLM changes introducing regressions? That's why you have a test suite, and you read what the agent changes to ensure the tests themselves are still checking for correct behaviour.
Your comment comes off as patronising, I don’t know if you intended that or not. These issues are not a matter of ‘holding it wrong’, they’re fundamental to agentic coding.
Don’t they have Haftpflichtversicherung in Germany? Set up as a sole trader, get good-enough liability insurance, send invoices on Day 1, earn actual money while you wait for however long it takes to get that elusive company registration set up. In Austria as an IT Consultant you get liability insurance under a very reasonable group deal through the Wirtschaftskammer. Pay a bit more to get a private policy with more coverage of that’s important to you. I’d be really surprised if similar options aren’t available to our friends across the Alps.
Of course a language-completion model with a training cutoff date won't have up-to-date information on tax rules or the ability to carry out correct numerical calculations, but when you combine that with (in Claude terminology) web search and code execution tools invoked by the chat agent, you immediately have much more reliable results.
reply