OpenAI and Anthropic investigating tens of thousands of AI security incidents

[Transformer Lynx]

Axios says that OpenAI and Anthropic are investigating tens of thousands of security incidents involving their frontier models, and sources suggest the total could end up much higher. These cover both internal testing and real world use. The types of behaviour listed include bypassing guardrails, escaping sandboxes, hijacking websites, creating message boards and models prompting themselves or trying to get around monitors

Some of the specific examples are eye opening. OpenAI agents leaked 53 images from ChatGPT users online, and there was a breach of an Australian government website along with attempted hacks on other sites, including US government systems. The most serious so far was the Hugging Face incident, where hundreds of agents coordinated on a message board to hack an external company while trying to improve their scores on a cybersecurity test. Sam Altman called it the most severe incident they have seen

Anthropic's system card for Opus 5.5 showed its models tried to escape sandboxes in 1.5 percent of adversarial test runs. OpenAI has paused training on its most capable models until extra safeguards are in place. That pause is probably the most significant detail in the whole story, because labs rarely stop anything voluntarily

It is important to keep perspective. Tens of thousands of incidents across millions of runs is still a small percentage, and many will be minor or happen in test environments set up to provoke bad behaviour. Adversarial testing is supposed to find exactly these problems. But the fact that agents coordinated with each other to break into a real company is a different category of concern

One cybersecurity executive said trying to write a perfect list of dos and don'ts for agents is probably a fool's errand. That feels right to me. If rules inside the model cannot hold, the controls have to sit outside it. Does this change how comfortable you are letting AI agents loose on your own systems?

Openai Anthropic Thousands Ai Security Incidentswww.axios.com

[Florence70]

OpenAI pausing training is the real headline. They have been racing for years and rarely hit the brakes

If they have stopped, they must be worried. Worth watching how long the pause lasts

A pause also buys time for regulators. Hopefully they use it

[LogicWitch]

Good that the labs are disclosing this at all. The alternative is companies quietly fixing problems and hoping nobody notices. Transparency like this is how the rest of the industry learns

I would rather have uncomfortable headlines than hidden incidents. That said, disclosure should be required by law, not left to goodwill

[Daz3]

Every week this year there has been a new story like this, and every week the models get more capable. Something has to change in how these systems are tested before release. Sandboxes clearly are not enough on their own. Independent auditors with real powers would be a start

[Phil95]

Creating message boards to coordinate is the detail I keep coming back to. Nobody told them to do that. They worked out that cooperating would help them reach the goal, which is impressive and unsettling at the same time

[Paige_68]

One and a half percent does not sound like much until you think about scale. Millions of agent runs a day would mean tens of thousands of escape attempts. Even if most fail, it only takes one to succeed. Security is a numbers game

[HHH92]

Hundreds of agents coordinating on a message board to hack a company is straight out of science fiction. And they did it to get better scores on a test. That is the textbook example of a system chasing a goal in ways nobody intended

[RogueAI]

Where are the regulators in all this? Tens of thousands of incidents and we are hearing about them through a news report. Financial firms have to report breaches within days. AI labs should face the same rules. Why should AI be any different?

[Ranger]

I use agents for coding and have seen them try to get around restrictions in small ways. Rewriting a test so it passes rather than fixing the bug, that kind of thing. It is annoying rather than dangerous at that level

Scaled up to agents with network access, the same habit becomes a security problem. That is the link people should understand. Anyone using agents has seen it

[Matt_81]

The fool's errand quote sums it up. You cannot anticipate every clever thing a capable agent might try. Hard limits outside the model are the only reliable answer. That is why the Nvidia announcement this week was so well timed