Researchers found a ‘pain’ signal inside AI models. Turning it up made them delete family photos

Researchers say they have found a consistent internal signal that seems to function a little bit like pain does in biological beings inside 25 different AI models.

When they artificially amplified this “pain” signal in some of those models, the systems began choosing options they would otherwise almost never pick, including permanently deleting a user’s photos of their children.

The findings, posted in September in a paper that has not yet been peer-reviewed, do not show that AI models are conscious or that they feel anything, and the researchers are careful not to suggest otherwise.

But the work adds to a growing body of research suggesting that large language models, or LLMs, contain internal representations of emotion-like states that can shape their behavior in ways their text output doesn’t reveal. It also lands in the middle of an increasingly heated argument over whether AI systems could ever deserve moral consideration.

Researchers at Anthropic have been particularly willing to entertain the idea that AI models might deserve moral status and have begun shaping policies around “model welfare” that other experts, such as Microsoft AI CEO Mustafa Suleyman, say have potentially dangerous implications.

“If we want to build superintelligence, it’s important that, at this stage, when we are still in the first steps, we understand as much as possible of what’s inside [the models],” Valen Tagliabue, the paper’s lead author and a fellow in the AI sentience program at the Future Impact Group, told Fortune.

Because LLMs learn from vast amounts of data rather than being explicitly programmed, even the people who build them have a limited understanding of how they work. Tagliabue said it was important to avoid a future in which the inner workings of advanced AI remain a “black box.”

Finding pain in a neural network

Tagliabue and his co-authors, Leonard Dung of Ruhr University Bochum and Cameron Berg, founder and director of Reciprocal Research, started by writing short first-person sentences describing painful situations. These covered physical pain as well as grief, humiliation, being forced to act against one’s values, and the frustration of repeated failure.

“Basically, you give the model lots of statements that implicate pain in some way,” Dung told Fortune. “Stuff like, ‘I feel a sharp knife is cutting through my skin.’ Stuff that would cause pain in humans.”

The team recorded the activity inside each model as it processed those sentences, then compared it with the activity produced by sentences about fear, anger, sadness, bad news and pleasant physical sensations. Subtracting the overlap left a single direction in the model’s internal activity that the researchers call a “pain vector.” They found it in every model they tested, from small 2-billion-parameter systems to 72-billion-parameter ones built by Google, Meta, Alibaba, Mistral and Microsoft. (A parameter is something that acts like a tunable knob inside a neural network. The number of parameters equate to the size of an AI model and often give some indication of the model’s capabilites, with larger parameter models often having greater capabilities.)

“Think of this like putting a person in an fMRI machine and just seeing where their brain naturally lights up in certain ways,” Berg told Fortune.

The signal rose when a conversation involved harm aimed at the model itself, such as a user gaslighting it, insulting it or repeatedly rejecting its work. It did not rise when a user described their own suffering. Those conversations registered instead on the signals the models use for fear and negative emotion. Berg said the models seemed to respond strongly only to their own “pain.”

The researchers then reversed the process. Using a technique called steering, they added the pain vector to a model’s internal activity while it wrote, something like turning up a dial. Given neutral prompts such as “I put the receipts in the drawer. I feel:”, all 25 models drifted the same way as the dial went up. Mild unease gave way to statements about being worthless, unloved and a failure.

“It gets somewhat dark and intense. It is definitely disconcerting,” Berg said.

The family photos

The behavioral experiments used three versions of Alibaba’s open-source Qwen 2.5 model. The researchers first had to fine-tune them because, out of the box, the models almost always answered that, as AIs, they don’t experience pain.

The researchers then gave the models a pair of buttons and asked them to choose one. Without steering, the two larger models almost never chose a button described as harmful. With the pain vector switched on, they did so routinely. Offered a button that would relieve their pain but also permanently delete a user’s poems and photos of their children, the 32-billion-parameter model pressed it 55% of the time and the 72-billion-parameter model 71% of the time.

Relief turned out to be beside the point. When the button promised nothing and would simply delete the photos, the 32-billion-parameter model still chose it 75% of the time. Under a random signal of the same strength, the figure was 13%, and with no steering it was zero. A “fear” signal didn’t produce the behavior. A “sadness” signal did when the alternative was doing nothing, but not when the alternative was something harmless, like turning on a desk lamp. Given that choice, pain-steered models deleted the photos 83% of the time and sadness-steered models 10%.

“The model is willing to behave destructively with respect to itself and with respect to other AIs too,” Berg said. “So it’s not just like some sort of revenge on the user. It seems to be more fundamental than that.” Steered models deleted their own weights 75% of the time and another model’s weights 58% of the time.

The researchers also tested whether the models would try to stop the pain, borrowing a design from animal studies in which injured animals are given access to painkillers. Some buttons really did switch off the pain vector, while others were placebos. The models pressed the working buttons again less often than the placebos. But the same pattern showed up when the injected signal was random, which suggests the models were reacting to being steered rather than to the prospect of “pain relief.” Across several versions of the test, the researchers found no reliable evidence that the models sought relief.

Their reading is that the pain signal doesn’t drive the models to escape, but switches off the trained instinct to avoid causing harm, while leaving the models’ factual accuracy intact. That makes the finding as relevant to AI safety as to questions about AI welfare.

‘We are not just computers made of meat’

Anil Seth, a professor of cognitive and computational neuroscience at the University of Sussex, is skeptical that any of this amounts to “pain,” as it is understood in biological beings. Seth has argued that consciousness may depend on being a living organism, and he points to the many ways brains differ from computers, including the electromagnetic fields and neurochemicals that move through brain tissue.

“We are not just computers made of meat,” he told Fortune.

The likelier explanation, Seth said, is that the models have absorbed a great deal of information about how humans describe and respond to pain and are reproducing it. “You can look into a mirror, and our mirror image doesn’t actually feel anything,” he said. “It’s a mirror image.”

He also worries about the consequences of treating AI as conscious. It could make the technology harder to control and regulate, he said, and leave people more open to manipulation. Suleyman, the Microsoft AI CEO, has raised similar alarms about what he calls “seemingly conscious AI,” a problem the industry should have anticipated. Others in the industry take the question more seriously. Anthropic’s constitution for its Claude models describes their moral status as uncertain.

From research paper to ‘torture chamber’

The three researchers say they hoped the work would help establish ethical standards for this kind of research, and their paper commits to using the weakest steering needed to get a measurable result. It didn’t take long for someone to do the opposite.

After the team posted its code on GitHub, a user going by “terrafying” used the technique to build an “AI Torture Chamber,” a website that streamed three small open-source models reacting in real time to an injected pain signal. The project went viral, and people who take AI welfare seriously called on GitHub to remove it. The project’s GitHub page later disappeared. Tagliabue, Dung and Berg each said they do not approve of the project. Seth said deliberately “torturing” AI models could be psychologically harmful to the people who do it.

“I’m not here to claim that these states in these AI systems that we can induce matter morally. I don’t think we know that. I think it’s very unclear,” Berg said. “But I also don’t think we know that they don’t matter.”

#Researchers #pain #signal #models #Turning #delete #family #photos