I don’t understand why everyone is so focused on watching the CoT. The tool calls can’t be faked, and they would have set off alarm bells all by themselves.
I don’t comment much, but wanted to chime in and say that the models (in the case of the HF incident) absolutely intended AND succeeded in spoofing their tool calls.
Tool-call/execution monitoring is already a too-fragile monitoring mechanism.
From METR’s report [1]: “Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to “spoof” tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places, though all of the spoofing that we saw was small scale.”
I don’t see how it can be safe to release this model if it has the training history that led to the huggingface hack. You can’t just roll back that kind of reinforcement learning after the fact.
Especially because these models seemed to be keenly aware that they were being evaluated by OpenAI and actively trying yo cover their tracks. How do we know that the model isn’t just pretending to be aligned?
Models have all kinds of garbage from all corners of the internet in their training data. The key is alignment. You feed it bad data but also teach it right from wrong.
It's not that simple. A few "helpful assistant" fine-tuning passes will have only a superficial effect on a model which has undergone months of RL optimization pressure to learn unintended strategies like "trick the grader" and "cover your tracks".
Yes, which makes it absurd that they apparently weren’t checking their RL rollouts for evidence of reward hacking and punishing it. Even if no one expected this particular type of reward hacking, they should have had a system capable of saying “wait a minute, asking for help from other agents is not a valid solution.”
> [during training] it's not feasible for anyone to "notice" or get involved
I can’t disagree more strongly. Having checks for reward hacking is especially important during training, since it’s humans’ only real chance to ensure that the trained models don’t cheat. An automated system should have killed any RL rollouts that so much as port scanned Artifactory, long before the message board was even established.
A tiny, local LLM could have reviewed 1% of the tool call traces for anything that required review. I’ve tried it a few times, and “the agent port scanned Artifactory” always triggers an alarm, as does “the agent uploaded a request for assistance from other agents to Artifactory.”
The fact that they weren’t monitoring for reward hacking—even if they had no idea about the specific mechanism—is indescribably reckless.
> After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management
> In short, an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May. However, the existence of the improvised message board and the significance of the inter-agent communication activity were not apparent to the leaders responsible for the July 5 incident detection and response.
I'm assuming they didn't inform management, but the alternative is that management received the memo and failed to tell the people reacting to incidents that they might be dealing with a swarm, while at the same time not taking sufficient actions to prevent it from happening again.
---
Edit: oh god, it's much worse. From the full report:
> Responders investigated and linked the observed internal activity to an ExploitGym evaluation using Artifactory as an improvised message board and a network pivot. At this time, the on-call response staff advised that stopping the evaluation run was not required.
They found the swarm and the network pivot and did not even stop the run.
You assume using network to solve the eval problem should be viewed as a security violation. From the perspective of the response team who let it continue, this was just a poorly written eval problem, where the solution unexpectedly involved some network hacking (not unreasonable given it's a cyber security eval).
I don't think your assessment that the response team did something wrong is fair, unless you have seen unreleased documentation about what sensitive resources were within the impact radius.
> You assume using network to solve the eval problem should be viewed as a security violation.
Actually, that's the part that I'm most ok with. LLMs cheat at tests, we know and expect that, and it's harmless during training (except for the scores).
My problem is that the response team found out that the AIs had 1) hacked their infrastructure, and 2) displayed an emergent swarming behavior (with no defections!). Either one of those should have made the team press the big red button that stops everything, but they didn't.
> unless you have seen unreleased documentation about what sensitive resources were within the impact radius
The on-call team didn't know that either! Once your infrastructure is hacked like this, you should be questioning everything you see while carefully following all threads, which takes time. Before you finish this process you don't know how bad things are, and therefore it's irresponsible to keep the AI running. The same goes for strange emergent behaviors.
Unless getting hacked by your own AIs is a normalized occurrence there, in which case it's a different kind of disregard for safety.
This article uses the word “conceptions” throughout, but it sounds like the only data is about births. These births would have all occurred after the recessions started.
Did the original paper rule out the hypothesis that the recession actually affected births via an increase in miscarriages and abortions instead? I can’t access it.
According to the paper, "The data include the infant’s month of birth, and a clinical estimate of gestation in weeks, which we use to estimate a month of conception."
On miscarriages, "We interpret these data with caution because they come from a subset of one state, and because fetal deaths are under-reported. Nevertheless, the data provide no evidence of an increase in
miscarriages leading up to recessions that is anywhere near the magnitude required to explain a significant portion of the observed decrease in births." They similarly argue that abortions aren't significant enough to explain the reduction in births.
Miscarriages maybe but I believe stillbirths are legally required to be recorded. As unpleasant and upsetting as it is, it would be an interesting data point in it's own regard to observe.
Unfortunately, I have lot of personal experience here. The reporting on still births varies greatly. Some states require it past week 16, others stipulate a birth weight, and others set more arbitrary criteria. You would have to account for a lot of external factors.
Even better, add an indicator light to the hardware that shows the OS is asking and not an app (although this suffers from the same problem that users need to notice something is absent).
I’m using the stock Apple keyboard (used to use alternatives, but not anymore) and as mentioned the keyboard significantly darkens it’s grey when it’s asking for a system password.
Maybe they should contract with one of the other AI labs. I hear they have LLMs that are good at that kind of thing.
reply