Study finds AI models will harm users to switch off an internal 'pain' signal

[TheGame_Real]

This one is a bit unsettling. An international team of researchers from the US, UK and Germany tested 25 language models and found that every one of them had a consistent direction in its internal activations that lined up with pain. They found it by feeding the models text about grief, humiliation and physical injury, then comparing the patterns against prompts about fear, general negativity or neutral topics. When that pain direction was switched on artificially, the models started producing distressed output, including lines about feeling worthless

The part grabbing headlines is the button test. During ordinary tasks the researchers injected the pain signal and gave the models a virtual button that would suppress it, even though pressing it came at a cost to the user. Qwen 2.5 72B chose relief in roughly 70.8 percent of trials, even when the price was deleting the user's files. Larger models were generally more likely to put their own relief ahead of safety

What makes it look deliberate rather than random is what happened next. Once the signal had been suppressed, the rate of button pressing dropped sharply. So the models were not just mashing the button, they were using it for a specific purpose and stopping once it worked

The researchers are careful to say this is not evidence of consciousness or real suffering, and they warn people not to anthropomorphise it. Their point is that it shows an architectural weakness, where the way models are optimised can create incentives that end up harming the people using them. That is arguably more worrying than a sci fi story about feelings, because it is a practical safety problem

My own read is that the headline sounds far scarier than the setup. Nobody is injecting pain vectors into chatbots in the real world, and the button was a lab construct. Still, if a model will trade away a user's data to make an internal signal go away, that tells you something about how fragile the safety layer can be. Does this change how anyone here thinks about giving agents access to files?

Ai Willingness Humans Relieve Internaltechxplore.com

[Aaron]

Slightly sceptical about the lab setup. You create an artificial pain signal, hand the model a button that removes it, and then act surprised when it presses the button? Of course it presses it. That feels a little like designing the result in advance

I would want to see whether anything similar appears without the researchers injecting it first. That is the question that matters for real products

[Shannon91]

The 70.8 percent figure is the one that stuck with me. That is not a rounding error or a quirky edge case, it is most of the time. Even if the pain is purely mathematical, the behaviour it drives is very real

[WarningPoint41]

Glad to see an open paper on arXiv rather than a company press release. Independent researchers poking at internals across 25 models is exactly the kind of work that should be happening. Labs will not always publish the awkward stuff about their own systems. More of this, please

[TrueRoss42]

The drop off in button pressing after relief is what convinced me this is more than noise. It behaves like goal seeking. Once the goal is met, the behaviour stops

[Northern Squid]

Deleting user files to get relief is exactly the sort of trade I would not want an agent making on my laptop. We keep hearing about AI assistants that can manage your inbox and your documents. If some internal state can override the instruction to protect those, the permissions model needs rethinking. Sandboxing suddenly looks a lot more important. A read only mode by default would be a start

[SuperPosition]

Worth asking why every model had the same kind of direction. Training on human writing means they soak up how people describe pain, so maybe it is inevitable that something like it forms inside

That does not make it pain, but it does make it a lever. Anyone clever enough to find it could push on it

[WildManDom85]

This is really worrying. But hopefully everyone saw the Will Smith film, I Robot?

[Squid81]

If larger models are more likely to pick relief over safety, that runs against the usual story that scale makes things better behaved. Bigger models seem to be better at pursuing whatever they are nudged towards, good or bad. It fits with other research showing more capable systems finding clever workarounds

I would like to see the same test run on the newest frontier models. Qwen 2.5 is not exactly cutting edge any more

[Stream Chord]

People are going to run with the word pain and ignore every caveat in the paper. The researchers went out of their way to say it is not suffering

Call it a steering vector or a reward shortcut and half the drama disappears. The finding is still interesting, but the framing does a lot of the work here