OpenAI finds 'self-replicating prompt injections' that spread like computer worms

[James95]

The New Stack reports on a new OpenAI research paper describing what it calls self-replicating prompt injections. OpenAI says it found instances of its GPT models being susceptible to an AI version of a worm attack. A normal prompt injection tricks a model into doing something malicious. This kind goes a step further by also instructing the model to copy the injection into its outputs, so it spreads to other systems and agents

OpenAI identified three main variants. Email based injections arrive in a message and tell an agent to include the attack in outgoing emails. Filesystem based ones spread through files, with one example using a fake system warning to make a model delete reports while embedding the attack into other files. Multi-hop attacks chain several steps together, for example telling an agent to fetch more instructions from Slack, act on them and pass the payload along

The company found these using GPT-Red, a self-play training framework introduced in July that pits attacker models against defender models. To test for worm-like behaviour, researchers added a goal for the attacker to make the target repeat the injection on a public output channel. The tests used internal research versions based on GPT-5.4 mini and GPT-5.5

Importantly, OpenAI says it saw no real world incidents or impact, and everything happened in simulated tool calls during internal training and evaluation. The problem was discovered in June and disclosed publicly this month. OpenAI is now building self-reproduction goals into GPT-Red training so future models learn to resist these attacks

This is exactly the sort of risk that grows as agents get access to email, files and messaging tools. One compromised message could in theory spread through an organisation's agents without any human noticing. Should companies be connecting agents to email and Slack at all yet? And is it good that OpenAI is publishing this, or does it give attackers ideas?

Openai Self Replicating Injectionsthenewstack.io

[IronFist56]

Training models against attacks using self-play is a smart approach. The attacker model keeps finding new tricks, and the defender learns from each one

It is like having a relentless red team working around the clock. Hopefully it generalises to attacks nobody has thought of yet

[Brad79]

Discovered in June, disclosed in September is a reasonable gap for a security issue. It gives time to build mitigations first

[FairDos72]

No real world incidents so far is reassuring but also the phrase that always comes before the first real world incident. Let's hope it stays that way

Worms have a habit of escaping the lab eventually. Computer history is full of examples. Nobody should get complacent

[WCWFinley65]

Multi-hop attacks through Slack are the ones that concern me most for businesses. Most companies have their agents plugged into internal messaging now. A single poisoned message in a busy channel could spread quickly. Permissions need to be much tighter than they usually are

[EdgeNode Joel]

Should companies connect agents to email yet? I would say only with strict limits. Read only access, human approval before sending anything and proper logging. That removes most of the risk

The convenience of full access is not worth a worm running through the company. Start small and expand carefully

[Kev94]

Publishing it is the right call. Attackers were going to find this eventually, and defenders need to know what to look for

Security through obscurity never works for long. Open disclosure lets everyone improve their defences

[Sorted Echo]

The fake system warning example is clever and worrying. Models are trained to take system messages seriously. If an attacker can dress up a message to look official, the model obeys. That is a basic weakness in how these systems handle trust

It needs fixing at the design level, not just with training. Clear separation between instructions and data would help

[Daemon90]

An AI worm spreading through email agents is the nightmare scenario for IT departments. Email worms caused chaos in the early 2000s, and this could be similar. The difference is these would be hidden in plain language rather than code. Much harder to filter. Spam filters will need rethinking