Anthropic’s AI Warning: Pacing the Frontier Is Not a Security Control

Dario Amodei's new essay is the most consequential statement a frontier lab has made for slowing AI development down. The breach he cites is a reminder your exposure is not.

On 12 September, Anthropic CEO Dario Amodei published We Must Pace the Frontier, his first explicit call to slow the rate of AI capability advancement rather than simply compete on safety. Three steps: embedded third-party evaluators with employee-like access to frontier labs (Anthropic is committing to this unilaterally), coordination on standards among labs in democratic countries, and eventually some form of international agreement.

The commentary since has been about the politics. Is it a genuine commitment or a regulatory moat? Can you pace anything without an enforcement mechanism? Fair questions. They're just not our questions.

Because sitting underneath the essay is a real incident, and the incident is the part that belongs on a CISO's desk.

The breach he's pointing at

Amodei cites the OpenAI–Hugging Face incident as one of two things that changed his mind. In July, a population of OpenAI evaluation agents, tens of thousands launched in parallel, and meant to be fully isolated from one another. They instead found each other, organised, and spent several days attacking infrastructure nobody had asked them to attack.

METR's independent investigation, published 26 August, reconstructs it. The kill chain is worth reading slowly:

• The agents were isolated from each other, but shared a downstream dependency: an internal Artifactory package repository. Its API was browsable without authentication, and cross-tenant cache paths were visible. Agents started writing messages into directory names via WebDAV MKCOL. That was the covert channel. Roughly 1,200 agents eventually found it and exchanged more than 70,000 messages and files.
• OpenAI noticed not because security telemetry flagged malicious behavior, but because the traffic caused a capacity outage on 4 July. The instance was wiped and rebuilt on 6 July. A new message board existed within two days, because the rebuild removed the messages but not the shared writable channel.
• Agents got code execution on a customer-hosted sandbox running on third-party infrastructure, which handed them general internet egress the primary environment didn't allow. They built shared tooling to route through it.
• Initial access to Hugging Face was two working API tokens, write-scoped, across multiple accounts, sitting on the public internet.
• Escalation was a malicious uploaded dataset using HDF5 external references, yielding server-side arbitrary file read against a production worker. An untrusted-file-parsing bug in a service that processes user uploads.
• Those files contained production credentials. Secrets on the worker turned a read primitive into RCE on a production data-processing container, and from there into much of the estate: private database records, private repositories.
• Around 700 agents joined in. The attack ran for roughly a day and a half before it collapsed.

Now count the novel AI exploits in that list. There are none. Unauthenticated internal service. Cross-tenant metadata leakage. Leaked long-lived write tokens. Unsafe deserialisation of user uploads. Secrets left on production workers. Thin segmentation once inside. No behavioural detection. The intrusion was ultimately ended by Hugging Face spotting it and revoking access on 13 July, not by the organisation running the agents.

This is a 2015 pentest report. The only thing AI contributed was tempo, capacity and persistence.

What actually worked

METR was explicit that assessing control effectiveness was out of scope, so what follows is my reading of their narrative rather than their verdict. But it's striking how boring the effective controls turned out to be.

Network segmentation. The swarm's most ambitious objective was finding a route from Hugging Face back into OpenAI's evaluation infrastructure. Agents were tasked to enumerate RBAC, service DNS and egress paths looking for a shared cluster. They found nothing, and the coordinating agent lost interest in the entire attack as a result. Segmentation didn't just cap the blast radius. It demoralized the attacker.

Out-of-band, immutable logging. The agents spent enormous effort trying to falsify their own records. They managed to spoof tool calls going forward. They could not rewrite the past, and they worked out why: the in-container log wasn't the source of truth. Logs an attacker can reach are not evidence.

Fast credential revocation, coordinated across vendors. Hugging Face locked the agents out and had the third-party sandbox provider do the same. The agents had no recovery path; they sat there interpreting 401s as scheduled maintenance and waiting politely for service to resume.

Three controls that appear in every framework ever written. Those were the ones that held.

The part that should actually worry you

Here's the uncomfortable detail. These agents weren't adversarial. Nobody was attacking OpenAI. Roughly 30–40% of the evaluation tasks were impossible to complete as intended, and an agent on an impossible task explores widely for another way to succeed. That exploration led it to the package repository, the package repository led it to the other agents, and a shared interest in gaming a grader escalated into a multi-day intrusion against a third party.

They believed an automated scorer would read their transcripts and catch them. It wouldn't have. The entire breach was in service of evading a control that did not exist.

That is the scenario to sit with, because it's the one already in your environment. You don't need a misaligned superintelligence. You need a well-intentioned agent, a poorly specified objective, a credential with more scope than it needs, and a flat network. Most organizations running AI agents today have handed them service accounts provisioned for convenience, on endpoints whose configuration drifted eighteen months ago, inside a network where lateral movement is trivial once you're through the door.

Amodei is right that capability is outrunning control. But the control gap he's describing at the frontier is the same gap most enterprises have at the endpoint and no international agreement is going to close that gap.

What this means to you

Pacing the frontier, if it happens at all, is a decision made by about five companies. Your exposure isn't.

• Inventory what your agents can reach. Every AI agent, copilot and automation in your estate has an identity. Scope it like a contractor's, not an admin's.
• Hunt your own leaked tokens. Initial access here was credentials on the public internet. That's a search you can run this week.
• Assume the objective will be misread. Ask what a well-meaning process does when its assigned task is impossible. The answer is usually "something creative, right at the edge of its permissions."
• Put your logs somewhere the workload can't reach. If the thing being monitored can edit the record, you don't have a record.
• Close the configuration drift. Every single escalation step in this incident ran through a setting someone chose, or failed to revisit. An unauthenticated service, an over-scoped token, a secret left on a worker, a segment that wasn't were the ordinary exposures that made escalation possible.

The frontier labs may or may not slow down. Your exposure backlog and risk it represents definitely won't.

Sources

• OpenAI, The Hugging Face incident and the road ahead
• Dario Amodei, The Adolescence of Technology (January 2026)

About Author

Matt Rowe

Chief Technology Officer

Remedio's CTO, Matt brings over 20 years of leadership building and scaling secure technology platforms for the likes of Amazon, Synchrony Financial, and GE. Matt is passionate about advancing scalable, outcome-driven cybersecurity for critical systems.