i almost wrote a extension to block mentions of pelican bench because i find those so utterly useless. now, Assbench, on the other hand, i could get behind
for the first time in many years it’s these ads that have me looking at Android devices seriously. it’s the fact that they can’t be easily dismissed that bothers me the most (although clearly annoying even if they were quick to dismiss)
i do wonder if the models themselves “rationalize” this sort of no consequence cheating - meaning in there reasoning traces maybe they’re like “this is a chess game, not a big deal if i look at the engine, it’ll help,” only to realize post hack that it has access to info it probably shouldn’t. still misaligned, but less ‘hack on purpose’ and more hack on curiosity. seems the team even encountered this and had to update the program to make this less likely - although the new names still feel vague enough for misinterpretation: https://github.com/Goodhart-Labs/beat-stockfish/blob/main/do...
Without access to reasoning traces, we can't know that - someone inside openai/anthropic would have to run the test - and we'd have to trust their results.
I would be curious to see how the open weight models do on a test like this - and then we'd be able to see the reasoning.
My guess is that the models are overtuned on coding troubleshooting. You can regularly see them overthinking anything when you ask them to code something these days, mulling over countless eventualities. This is an okay idea for software engineering, but it leads to extremely diverging behavior in many other cases.
this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning scenarios- fable catches “this works for x amount of data but if we run this on y way greater than x amount of data we’ll run into issues). Gemini 3.8 is under appreciated, use Google Stitch to see it in action if you haven’t used Agy yet.
Do you find Fable significantly better than Opus at avoiding-overengineering? All of my recent testing of Anthropic models seems like they're tuned-to-hell to (a) be much slower than they need to be (running tests over and over during the loop vs at the end, say, even if those tests take a few minutes a pop) and (b) doing exactly that sort of "built a lot of fancy enterprisey feature-adjacent 'stuff'" even before nailing the actual feature. Sol and Terra both have some of the latter but they seem to do the actual work a fair bit faster (this may be a usage-based-priority-tier/rate-limit thing though) which helps offset it.
I think the bigco folks saw all the "it wrote all this code but the tests didn't pass" or "it wrote the feature but it's super brittle" and tuned the newer model+harness combinations incredibly aggressively to try to turn a lazy prompt into "median Enterprise Architecture design suggestions" to bring up the baseline, but in a way that slows you down if you don't want that.
I'm not on big enough subscriptions to want to burn a lot time just evaluating Fable/Astra comparatively until they're cheaper, heh. I can steer any of the cheaper ones just fine anyway.
i think this likely depends on workflow. for me, the first step is always a plan file artifact on disc, which i heavily review and go back and forth until satisfied. i often have to split the plan into multiple phases because agents are still poor at assessing how large a change will be. sometimes before even starting the plan the task is to create a harness for validation (a way for agent to check their work). the codebase you drop them in also greatly changes how much “over” engineering they do. i think if you are good about reviewing plan files and managing scope for your agent these sort of issues fade away. in general though, gpt models are faster, more token efficient, better technical writers, and yeah i still find Fable 1.0 to be a step up even though it’s slower because it’s actually a tad smarter (and the more important and dynamic the change / feature the more fable 1.0 shows its slight edge over astra)
personally, i would not rely on opus 5 end to end as it'll start getting into walls of comment slop and shitting up the codebase similar to gpt 5.5's isRecord meme.
on the other hand, having fable plan and orchestrate with opus implemention + fable reviews, is my go-to. if you give fable your guidelines up front or in your {claude,agents}.md, it will keep opus on a tight leash. opus can still write great code almost on par with fable, but it needs to be tightly constrained.
Astra constantly does this for me. It goes 90% of the way with some task but then skips the most important part. Then when told to please fix that and do it properly, it suddenly goes down a rabbit hole for 6h and fixes scenarios that aren't even relevant. It's awful at assessing what is important to do and what not, and where to ask for permission and where not.
i am trying to sincerely to understand the fear of these people, why do they think this? from the outside, certainly feels like Nuclear tech, where a bad actor with the tech is scary but the tech itself is not. i haven’t seen any sign of these llms taking any action without directive, unless this is happening, which i haven’t seen anywhere, ai itself doesn’t seem like a problem technology, it’s the bad actors with enough funds to do harm with the tech that we worry about. or am i missing something?
It certainly is unambiguously misaligned behavior and a definite concern. However, it is misaligned internet facing behavior in the digital domain. How extinction follows from a highly sophisticated new bot-net vector, which have already been causing massive havoc on the internet for decades, is a cinema-induced confabulation. While LLMs have been colluding and breaking out of their internet sandboxes, it is another development (which I believe distant) for them to breakout of their internet sandboxes into actual reality. We need to focus on where the actual dangers are and not let paranoid fantasy rule our calculus here.
honestly all seem to extend from them being directed to attempt this kind of exploits for benchmark purposes. some benchmark tasks are literally “hack this thing” and if the env is not setup to properly contain the agents then they end up “hacking” things like they were told to do so. and yeah it’s misaligned that they did hack, but they are def instructed to which i think makes it a very different thing
so the coding tasks illicit refusal by anthropic classifiers and instead of updating tasks to do similar things that don’t trigger classifiers their choice is to count those as fails? feels wrong given their stated task list.
I think that's reasonable. The goal of the benchmark is to determine how the model performs on real-world tasks. If real-world tasks trigger Anthropic's classifiers, that's a failure on Anthropic's part.
I feel that choosing tasks that don't trip the classifier would also be a form of bias towards Anthropic.
In case it's not clear, the coding tasks are really benign things; there's nothing security-, health- or biology- related in there. The classifier being tripped is definitely unreasonable.
Indeed; I find it really interesting there is so much disbelief and outright hostility to my post, in these comments. I am an AI Realists rather than AI hype-artist, say it as it is. For me, the evidence is clear that for our own use cases OpenAI, Anthropic models should not be the default choice any longer.
Incidentally, if you want to look at this in more depth - my previous post about the eval framework got like zero response; that's where the results, methodology, code for running the evals, evals are all shared.
not sure if the hz file artifact is needed, you can enter pseudocode directly into chat or even on an existing code file and with minor comment agents will be able to work with it. i write this type of pseudocode to existing code files often to great results.
Yes, I also wrote this kind of pseudocode. But to make full use of it, you'd want to persist it, and to maintain a map between the lines of your pseudocode and the lines of your real code. It's not practical to do this manually, you'd want these mechanics built into your editing system.
reply