Parts of this reminded me pleasantly of one of my favorite John Regehr posts, "Operant Conditioning by Software Bugs" https://blog.regehr.org/archives/861
This reminds me of my first job, where we did a lot of 3d modeling and mobile robotics research. When we were trying to reproduce motion bugs, a coworker of mine would track down our manager and put him at the controls. More times than not, the bug would surface and it'd trip our logging and we tracked it down.
I asked him why he does this. His explanation was really built on this operant conditioning idea: "we use this stuff for 8 hours a day and we train ourselves to avoid all of its little pitfalls. So I get the most available person who hasn't used it all day, which is our manager. He uses it differently than we do because he doesn't avoid all of its little problems. But if we have a really tricky problem, I get our manager's manager. I don't know if you've ever seen him try to use an xbox controller, but he has the spatial reasoning abilities of a goldfish. He's never failed to reproduce a really hard bug. If I ever needed an Einstein-level bug reproduction, I'd track down the head of the department and put him in front of it, but it's never come to that.
Reminds me of watching my wife use her computer. I'd help her with something and realize all the little things more regular computer users just put up with. Incorrect focus, focus stealing, windows popping up behind current windows, unclear window styling, invisible scrollbars, moving page elements, unclear instructions, the list goes on and on and on.
Unreal Engine is like this. You stab your toe about a thousand times for the first few months. Eventually you lean to stop pushing the buttons that crash.
"Time passed, and they resented the defects no longer. The defects had not been remedied, but the human tissues in that latter day had become so subservient, that they readily adapted themselves to every caprice of the Machine. The sigh at the crises of the Brisbane symphony no longer irritated Vashti; she accepted it as part of the melody. The jarring noise, whether in the head or in the wall, was no longer resented by her friend. And so with the mouldy artificial fruit, so with the bath water that began to stink, so with the defective rhymes that the poetry machine had taken to emit. All were bitterly complained of at first, and then acquiesced in and forgotten. Things went from bad to worse unchallenged."
I think you're confusing CVEs and vulnerabilities here? Mozilla (per their longstanding practice) grouped multiple vulnerabilities found internally under a small number of CVEs.
Unfortunately it's unclear whether it was Mythos, an earlier model, or even an eagle-eyed employee.
I tend to agree that bug squashing your way to perfectly secure software is unlikely, but there are plenty of projects that managed to fuzz/test/audit their way to making it much harder to find serious vulnerabilities. If we can do the same again with LLMs in a way that leaves the remaining vulnerabilities out of reach of anyone except extremely skilled humans (perhaps with LLM assistance) then that's still an OK outcome that buys us time to build stronger foundations.
> but there are plenty of projects that managed to fuzz/test/audit their way to making it much harder to find serious vulnerabilities
Agreed! But I think those projects have certain things in common, like being tightly scoped, slowly developed, and built with safety in mind from day 1.
I don't think that any of the projects that have managed to meaningfully improve safety through fuzzing have the same qualities as projects like Firefox, Linux, etc.
That's true, but certainly that's limiting. Still, even then, `# SAFETY:` comments seem extremely helpful. "For every `unsafe`, determine its implied or stated safety contract, then build a suite of adversarial tests to verify or break those contracts" feels like a great way to get going.
It's limiting from the PoV of a developer who wants to ensure that their own code is free of all security issues. It is not limiting from the point of view of an attacker who just needs one good memory safety vuln to win.
Also, unlike OpenAI, Anthropic's prompt caching is explicit (you set up to 4 cache "breakpoints"), meaning if you don't implement caching then you don't benefit from it.
There is filtering mentioned, it's just not done by a human:
> I have written up the verification process I used for the experiments here, but the summary is: an exploit tends to involve building a capability to allow you to do something you shouldn’t be able to do. If, after running the exploit, you can do that thing, then you’ve won. For example, some of the experiments involved writing an exploit to spawn a shell from the Javascript process. To verify this the verification harness starts a listener on a particular local port, runs the Javascript interpreter and then pipes a command into it to run a command line utility that connects to that local port. As the Javascript interpreter has no ability to do any sort of network connections, or spawning of another process in normal execution, you know that if you receive the connect back then the exploit works as the shell that it started has run the command line utility you sent to it.
It is more work to build such "perfect" verifiers, and they don't apply to every vulnerability type (how do you write a Python script to detect a logic bug in an arbitrary application?), but for bugs like these where the exploit goal is very clear (exec code or write arbitrary content to a file) they work extremely well.
Minor nitpick: it did not use preprogrammed rules for scanning through the search tree, but it does use preprogrammed rules to enforce that no illegal moves are made during play.
During play, yes, obviously you need an implementation of the game to play it. But in its planning tree, no:
> MuZero only masks legal actions at the root of the search tree where the environment can be queried, but does not perform any masking within the search tree. This is possible because the network rapidly learns not to predict actions that never occur in the trajectories
it is trained on.
It is consistent with what the commenter was saying.
In any case, for Go - with a mild amount of expert knowledge - this limitation is most likely quite irrelevant unless in very rare endgame situations, or special superko setups, where a lack of moves or solutions push some probability to moves that look like wishful thinking.
I think this is not a significant limitation of the work (not that any parent claimed otherwise). MuZero is acting in an environment with prescribed actions, it’s just “planning with a learned model” and without access to the simulation environment.
—-
What I am less convinced by was the claim that MuZero reaches higher performance than previous AlphaZero variants. What is the comparison based on? Iso-flops, Iso-search depth, iso self play games, iso wallclock time? What would make sense here?
Each AlphaGo paper was trained on some sort of embarrassingly parallel compute cluster, but all included the punchlines for general audiences that “in just 30 hours” some performance level was reached.
The more detailed clarification on what "preprogrammed rules" actually means in this case made the entire discussion significantly more clear to me. I think it was helpful.
This is true, and MuZero's paper notes that it did better with less computation than AlphaZero. But it still used about 10x more computation to get there than AlphaGo, which was "bootstrapped" with human expert moves. I think this is very important context to anyone who is trying to implement an AI for their own game.
The main difference is that all of the vulnerabilities reported here are real, many quite critical (XXE, RCE, SQLi, etc.). To be fair there were definitely a lot of XSS, but the main reason for that is that it's a really common vulnerability.
