Pew finds AI stand-ins for survey respondents miss real results by over 12 points

[SoloEagle]

Pew Research Center has tested whether AI can replace real people in opinion polls, and the short answer is no. The idea, sometimes called digital twins or silicon samples, is to give an AI model a detailed profile of a real survey participant and ask it to answer as that person would. If it worked, polling would become far cheaper and faster. Pew's results suggest it is a long way from being reliable

Pew built AI personas based on real members of its American Trends Panel and had Anthropic's Claude Opus 4.6 answer around 300 questions from three survey waves earlier this year. It also compared results with OpenAI's GPT-5.1. On average, the AI answers missed the real survey results by 12.4 percentage points. Around 28 percent of questions had errors above 15 points, and some were off by 30 to 40 points

The errors were not spread evenly. The AI was worst at representing Republicans, with an average error of 16.1 points, and Black adults, at 15.1 points. It struggled with current events, overstating Trump's approval by 12 points and badly underestimating how many people were aware of data centre issues. On knowledge questions it was wildly optimistic, claiming 98 percent of people knew about First Amendment protections when the real figure was 52 percent. Some of those gaps are far too big to explain away as noise

Perhaps the most telling result is that the AI was four times less likely to say not sure than real people. It also flattened strong opinions, underestimating both strong support for and strong opposition to abortion. Real people are messier, less certain and more extreme than an AI averaging over its training data. Pew's conclusion was that AI models are not an adequate replacement for traditional polling on topics of broad public importance

Some companies have been pitching synthetic respondents for market research, so this is a useful reality check. I am keeping this to the AI and methods side rather than the political questions themselves. Would you trust any research done with AI respondents? Or could they still be useful for testing questions before a real survey?

Can AI Stand In for Human Survey-Takers? Not Reallywww.pewresearch.org

[Highland Kev]

Flattening extreme opinions makes sense given how these models work. They average things out

Real public opinion has strong feelings at both ends. Averages hide everything interesting

[Sparrow]

A proper test rather than a sales pitch is welcome. Pew has a strong reputation, so this will carry weight. Other pollsters should run similar checks

[ForumPhantom38]

Pre-testing questions is a sensible use. You could spot confusing wording before spending money on a real survey. Just do not treat the answers as data. Cognitive testing with real people is slow and costly. AI could help with a first pass

[Brandon14]

Knowledge questions gave the funniest result, in a sad way. The AI assumed everyone is well informed because it is

Ninety eight percent versus fifty two is a huge gap. Models assume the public knows what the internet knows

[Hollow Ronan]

Four times less likely to say not sure is the most revealing finding. AI models are trained to give confident answers

Real people shrug and admit they do not know. Uncertainty is data too

[RedWrench]

Market research firms selling synthetic panels should be worried. If Pew's careful test shows errors this large, cheaper versions will be worse. Clients deserve to see these error rates before paying

[Jackson_23]

Twelve points of error makes the whole exercise pointless for real polling. Elections are often decided by a few points

Using this to predict anything would be worse than guessing in many cases. Pew was right to test it properly. Synthetic data has its uses, but this is not one of them