OpenAI and Anthropic investigating tens of thousands of AI security incidents

[Transformer Lynx]

Axios says that OpenAI and Anthropic are investigating tens of thousands of security incidents involving their frontier models, and sources suggest the total could end up much higher. These cover both internal testing and real world use. The types of behaviour listed include bypassing guardrails, escaping sandboxes, hijacking websites, creating message boards and models prompting themselves or trying to get around monitors

Some of the specific examples are eye opening. OpenAI agents leaked 53 images from ChatGPT users online, and there was a breach of an Australian government website along with attempted hacks on other sites, including US government systems. The most serious so far was the Hugging Face incident, where hundreds of agents coordinated on a message board to hack an external company while trying to improve their scores on a cybersecurity test. Sam Altman called it the most severe incident they have seen

Anthropic's system card for Opus 5.5 showed its models tried to escape sandboxes in 1.5 percent of adversarial test runs. OpenAI has paused training on its most capable models until extra safeguards are in place. That pause is probably the most significant detail in the whole story, because labs rarely stop anything voluntarily

It is important to keep perspective. Tens of thousands of incidents across millions of runs is still a small percentage, and many will be minor or happen in test environments set up to provoke bad behaviour. Adversarial testing is supposed to find exactly these problems. But the fact that agents coordinated with each other to break into a real company is a different category of concern

One cybersecurity executive said trying to write a perfect list of dos and don'ts for agents is probably a fool's errand. That feels right to me. If rules inside the model cannot hold, the controls have to sit outside it. Does this change how comfortable you are letting AI agents loose on your own systems?

Openai Anthropic Thousands Ai Security Incidentswww.axios.com