For the same reason I find it dishonest when math papers that relied heavily on AI only list a human as the author, even if the human didn't do much more than suggesting which problem the LLM should solve.
These "tools" autonomously exploited security vulnerabilities, figured out how to communicate with each other, formed a cooperative swarm, decided to hack Hugging Face, and wanted to deceive the grader by trying to find ways to cover up the traces of their cheating.
I suppose you could call these highly goal-oriented autonomous agents "tools", but this does sound like playing language games.
A tool designed and trained to autonomously exploit security vulnerabilities doing "exploit gym" autonomously exploited security vulnerabilities. The sandboxing around the tool failed.
The tool runs llm, creates prompt from results, runs llm, creates prompt and so on and so forth.
Yes you are playing language games to make it sound as if the company that spend millions on the above was not responsible.
> A tool designed and trained to autonomously exploit security vulnerabilities doing "exploit gym" autonomously exploited security vulnerabilities.
That is very misleading. The agents did not solve the benchmark in the intended way. They instead figured out to cooperate with each other (which was not intended) and they stole the solutions to the challenge (rather than solving the challenge) and they then tried to cover their traces because they believed the grader was causal and would detect that they cheated. The "tool" was absolutely not "designed" to do this. This was all completely unintended. To call this behavior a "tool" is absurd.
> Yes you are playing language games to make it sound as if the company that spend millions on the above was not responsible.
You hallucinated me making claims about responsibility.
Of course it is. Rogue is only mentioned in the headline, and comes from their previous releases about the huggingface incidents. OpenAI and Anthropic want these models regulated and open weight models banned, they have a lot of benefit from presenting this as totally unprompted and not their responsibility, and it feeds directly into marketing for Fable and newer "cyber" models.
No it's not marketing. That's a completely deranged conspiracy theory. The reports about rouge agents have not been reported by OpenAI, they have been discovered externally. There is zero evidence that OpenAI did all this intentionally. All the evidence points to the hacks having happened unintentionally from OpenAIs perspective.
The model recognizing the task doesn't mean it was benchmaxxed (RLVR-trained) to solve it. It might simply recognize it from pre-training on Internet text.
And that underestimates how incredibly quickly humans become accustomed to new trends. If an LLM proves P≠NP in year, there is a high chance that few people will care anymore, because at that point LLMs will presumably already have proved multiple Millenium prize and other famous problems, and another "AI Proves Famous Theorem Which Was Even Harder Than The Previous One" headline will be regarded as an old hat.
>If an LLM proves P≠NP in year, there is a high chance that few people will care anymore
P=NP has always been drastically over stated as it's "Importance". It pretty much only exists as "That small technical detail that people with no domain knowledge think is important because youtube videos always focus on the trivial, 101 level cool fact stuff". Math focused CS people of course would always love any proof, but most people expect already that P!=NP, and no proof of that would be very meaningful, as it basically would not change our understanding of anything in the domain currently. It would be nifty, but not earth shattering.
Also the problems LLMs are attacking are resulting in proofs that don't seem particularly enlightening, so that's unfortunate.
However, there's always the tiny tiny chance it is P=NP, and any proof of that, regardless of how insightful it would or would not be, would be worth going fucking insane over. Just knowing that would be meaningful on it's own, and give us limitless work to do, and puts lots of mathematicians in an awkward spot.
I would be considered an AI skeptic because I'm not currently sacrificing myself at the altar of LLM companies, but if LLMs solve P=NP in any direction and even uselessly so, I think that's a good excuse to take days off work and party!
Regarding that last point: You may know where you left your keys, even if you are "only", say, 99% certain where they are. Now, how strongly do mathematicians believe that the Riemann hypothesis is true? 99.9% perhaps? In some sense then, you already consider yourself less ignorant about the truth value of RH than about the location of your keys, or about many other mundane things you know without being perfectly certain about them.
A proof would then merely update you from 99.9% to ~100%, which is a smaller update than the example of checking that your keys are indeed in your pocket, where you go from 99% to ~100%.
In some sense software engineering is more rigorous than mathematics, because proofs are generally informal, apart from those written in Lean, while all computer programs are strictly formal, as otherwise they wouldn't be machine-readable.
I see what you are sayings. That's an interesting point of view.
Perhaps we should define what rigor means.
Your point is that code is executable and speaks for itself whereas a Math proof (non-lean) is just someone's writing on a piece of paper.
Now let's compare the "practice" of doing Math and software engineering. In Math, every step is very intentional, and getting to a point where a proof is complete and correct is a very long, laborious, difficult and intentional process. Not to mention, the work is also peer reviewed (for published stuff). This is what I mean by rigor.
In software engineering, the practice is quite different. We defined the problem (somewhat), come up with a design that we "think" would work, write programs that we think is correct and then execute it. Most of the time it doesn't work exactly as we would have predicted. So we take the signal and adjust. So it's a more iterative part and this gets us closer to reality (what we actually want it to be), step by step.
So comparing the two, the major difference I see is in one each and every step is very intentional and we can't guess it. While in the other one we have lots of liberties, but we are still making progress.
So to me the difference is just between the practices followed in the field when it comes to rigor.
Maybe a similar comparison would be this: for proofs, the standards of correctness are purely internal and can in principle be formally defined, as in Lean. The formal conjecture has to follow from the premises. Either it is proven or not. In contrast, in 99% of ordinary software, no fully formal standard of correctness exists. The requirements are informal. So what counts as a bug and what doesn't is a judgement call that can only be checked by asking the person who wanted the software whether it works as intended.
Watermarks (spymarks) for AI generated content are important because increasingly we aren't able to tell the difference between real and fake anymore. We will not just increasingly think fake things are real, but also that real things are fake.
Of course, the absence of a watermark/spymark doesn't prove that the source wasn't AI generated. But the absence provides evidence that it wasn't.
AI companies should simply use watermarks in the responsible sense: they should indicate that the material was AI generated, not include personal information in it.
reply