Even With AI, We Still Have To Poll Humans

AI chatbots are really good at writing like humans, although there are some obvious tells (it’s not X, it’s Y). So it’s not a leap to wonder if AI could be good at answering questions like a human, say, in a political survey.

Political polling is having a rough time. It’s almost impossible to get people to participate in a poll. For every 100 voters you ask to take a survey, only one or two actually go through with it and share their opinions. Pollsters often incentivize participants with gift cards to get them to complete the survey. For some subgroups of the population, the response rates are even worse.

For campaigns, that means polls are getting more expensive (we have to call and text more voters) and less reliable (we have to keep surveys open for longer periods of time), yet it’s still the best tool we have to understand what an electorate is thinking about a candidate, an issue, or a message.

That’s why it would be a lifesaver if AI could answer survey questions like a real human voter at a fraction of the cost.

A new report from researchers at Verasight demonstrates just how tall of an order that might actually be. “The best way to think about [current approaches],” they write, “is that they are predictions, not measurements of public opinion, and thus, are only as accurate as the underlying model of the response variable.”

AI, in other words, is a rearview mirror, looking backwards. It’s not a windshield. This makes sense, because LLMs are trained on mimicking written text. They “write” by probabilistically filling in the most likely next letter, word, sentence, and paragraph based on all of the other things it’s “read” before.

The report does share some glimmers of hope, like results on ballot tests and Trump approval ratings that come close to the tolerated errors of real surveys, but these are already widely available and track closely with partisan and demographic data. The AI survey engine struggles when current events come into play.

But even traditional polling – the benchmark in their research – is backward looking. It’s a snapshot in time of those polled that we use to make decisions about the future.

The science of survey research relies on the concept of the “representative sample”: if you ask one 40-something, college educated, married white man who is a Republican and lives in this ZIP code a question, we hope the chances are good the other voters who look like him also think the same thing.

That makes sense when our inputs are all the same as they were decades ago. Everyone watching the same TV news, reading the same newspaper, and going to the same church. But today, our media diet – the inputs – are as unique as we are and I’m skeptical that anyone – man or machine – can capture a sufficiently representative sample.

Imagine you’re going to a tailor to be fitted for a new suit, but the only way he can measure you is with the chains used in football to determine a first down. That’s where we are with polling in politics. Narrow margins, diverse subgroups, and fewer observations. Survey research on more volatile questions that haven’t reached some sort of equilibrium (like presidential job approval or attitudes about abortion) becomes more unpredictable.

For now, we’ll stick with the “devil we know” and rely on polling – asking human beings what they think – to guide our strategy.