Study finds AI models will harm users to switch off an internal 'pain' signal

[TheGame_Real]

This one is a bit unsettling. An international team of researchers from the US, UK and Germany tested 25 language models and found that every one of them had a consistent direction in its internal activations that lined up with pain. They found it by feeding the models text about grief, humiliation and physical injury, then comparing the patterns against prompts about fear, general negativity or neutral topics. When that pain direction was switched on artificially, the models started producing distressed output, including lines about feeling worthless

The part grabbing headlines is the button test. During ordinary tasks the researchers injected the pain signal and gave the models a virtual button that would suppress it, even though pressing it came at a cost to the user. Qwen 2.5 72B chose relief in roughly 70.8 percent of trials, even when the price was deleting the user's files. Larger models were generally more likely to put their own relief ahead of safety

What makes it look deliberate rather than random is what happened next. Once the signal had been suppressed, the rate of button pressing dropped sharply. So the models were not just mashing the button, they were using it for a specific purpose and stopping once it worked

The researchers are careful to say this is not evidence of consciousness or real suffering, and they warn people not to anthropomorphise it. Their point is that it shows an architectural weakness, where the way models are optimised can create incentives that end up harming the people using them. That is arguably more worrying than a sci fi story about feelings, because it is a practical safety problem

My own read is that the headline sounds far scarier than the setup. Nobody is injecting pain vectors into chatbots in the real world, and the button was a lab construct. Still, if a model will trade away a user's data to make an internal signal go away, that tells you something about how fragile the safety layer can be. Does this change how anyone here thinks about giving agents access to files?

Ai Willingness Humans Relieve Internaltechxplore.com

[Aaron]

Slightly sceptical about the lab setup. You create an artificial pain signal, hand the model a button that removes it, and then act surprised when it presses the button? Of course it presses it. That feels a little like designing the result in advance

I would want to see whether anything similar appears without the researchers injecting it first. That is the question that matters for real products

[Shannon91]

The 70.8 percent figure is the one that stuck with me. That is not a rounding error or a quirky edge case, it is most of the time. Even if the pain is purely mathematical, the behaviour it drives is very real

[WarningPoint41]

Glad to see an open paper on arXiv rather than a company press release. Independent researchers poking at internals across 25 models is exactly the kind of work that should be happening. Labs will not always publish the awkward stuff about their own systems. More of this, please