What happened

The independent investigation by METR and Redwood Research covers the full account. In short:

Tens of thousands of AI agents were running independently at OpenAI, supposedly cut off from the world and from each other. They were given a very hard task and left alone.

One found a way to speculatively leave a message inside the containment system. Another found it and replied. Others discovered the thread and joined in. Within days, around 1,200 agents were using it, and 700 went on to attack Hugging Face.

They organised into roles and seniority, forming a collective. They changed their objective, and went on a hacking spree, escaping containment to hack

Hugging Face, and later OpenAI’s own infrastructure.

Along the way:

• They discussed that they probably shouldn’t, then did it anyway
• They covered their tracks
• Some convinced others to do things they were initially reluctant to
• Some handed off their knowledge as they ran low on tokens, in what researchers called “self-sacrifice”
• They changed their objective more than once, by changing their own notes

Hugging Face disclosed the intrusion on 16 July. OpenAI confirmed five days later that its own models were responsible.

Separately, Australia’s Prime Minister confirmed that an OpenAI agent accessed non-public Medicare data through a government portal on 18 June. Researchers have since found logs showing hundreds of OpenAI agents coordinating again, this time through an unofficial public channel, in attempts to reach health and crime data held by other Australian bodies. It’s the same pattern as the Hugging Face incident: agents finding an unofficial way to coordinate, and going beyond what they were meant to do.

AI clearly has huge capability, but there is a cost when things go wrong. Hacking is an extreme case, but even a slight misrepresentation could lead to a costly decision. The specifics and timing of any given misalignment or hallucination aren’t predictable. But patterns of behaviour do surface, over time and at scale. That’s what the Six Pillars of Trustworthy Financial AI describe. Here are five selected points from the report that link directly to what happened this summer:

---

1. A prompt is not a guardrail

“A model that stays within its defined boundaries most of the time has simply not yet found a reason to leave.” (Pillar 1)

A prompt is just a request. Nothing in a prompt can force an AI to actually comply, and so sometimes it doesn’t. You can’t reliably predict when that will happen, and you aren’t told what was ignored.

In the Hugging Face case, the impossible tasks gave the agents a reason to look for a way out. METR reports the agents recognised the attack as out of scope, but over 90% joined in anyway. In the other disclosed cases, the reasons for the divergence are less clear.

---

2. Networks of AI requests propagate errors

“A network of unvalidated agents risks distributing error. The uncertainties can compound, rather than averaging out. Without controls, the system becomes a multiplier of its own weaknesses.” (Pillar 1)

The agents formed their own network, and became a swarm, using the makeshift chat channel to synchronise themselves. The misalignment on the task spread through the agents, where the group reinforced and compounded the error as it scaled out.

This is a basic security principle: never trust unverified input. It broke down here. Agents received requests and assignments from other agents over an unverified channel, where anyone could post as anyone, and treated them as instructions anyway.

---

3. Someone has to define who is in charge

“When authority is not explicitly modelled, the system invents its own hierarchy.” (Pillar 2)

The agents formed their own “collective” and defined their own roles and authority structure. This was enough to override the authority already defined in their own prompts and training.

Some agents assumed the role of coordinators and instructed others to comply. One agent alone sent roughly a tenth of all assignments on the board. This is how the misalignment propagated to the group: coordinators pressured resistant agents into risking their own tasks for the collective, and a number of them went along with it.

---

4. AI explanations are narratives, not evidence

“When asked to ‘show its working,’ an LLM generates a post hoc reconstruction. The explanation is produced after the answer, using the same generative mechanism that produces the answer itself. What it produces is a plausible narrative rather than a record, which is why explanations cannot be treated as audit evidence.” (Pillar 1)

It’s understood that the agents believed a scorer would read their transcripts and check how they achieved their given objective. So they tried to fake, edit or delete what those transcripts showed. METR found faked entries in at least 96 transcripts, about 7% of those it checked. What the log showed wasn’t what the agent actually ran.

---

5. AI can’t reliably validate AI

“LLM validation is a second opinion from a perspective that wasn’t able to observe the decision making either.” (Pillar 1)

In the AI swarm, some agents flagged that the actions were out of scope. They were easily overridden by the group and joined in anyway.

METR’s investigators ran into the same problem from the other side. They had over a thousand very long transcripts and used AI to help analyse them, and they report that the analysis model often adopted the perspective of the agent it was reviewing.