What To Do When AI Comes for Online Panels
Last month, I joined hundreds of other pollsters and researchers at the American Association for Public Opinion Research (AAPOR) conference in Los Angeles to discuss best practices, the future of our industry, and new methodological innovations. Unsurprisingly, this year’s conversation largely centered on the rapid progress of artificial intelligence, which has reshaped several longstanding debates. Among them was a topic we’re deeply invested in at BlueLabs: the tradeoffs between probability sampling and opt-in panels, where respondents volunteer or are selected nonrandomly.
Online nonprobability panels fill a critical role for modern pollsters. They’re cost-effective and flexible, allowing for design choices that don’t always translate across modes (such as embedded multimedia). But because these panels don’t sample randomly and pay respondents for each completion, they’re vulnerable to inattentive, low-effort (also called “satisficing”), or outright fraudulent responses. Pew Research Center has done excellent work documenting how these factors can produce unreliable estimates for adults under 30, Hispanic respondents, and other groups that are often underrepresented on panels — possibly because they are more likely to be allowed into a survey or paid at a higher rate. This incentivizes bot farms and fraudulent respondents to misrepresent themselves as members of these groups.
At BlueLabs, we see these online panels as one useful tool, not the be-all and end-all. Our Senior Director of Polling, Joy Wilke, has written about why multimodal surveys are still worth the investment — check it out here.
So, what’s the AI angle? In short, AI has supercharged fraudsters’ ability to game the system. Many classic bot-detection approaches relied on in-survey “trap” questions to catch satisficing or fraud. A common attention check might ask how often respondents do a certain activity, then flag anyone who claims they “never” breathe air. Others involve behavioral signals, such as flagging abnormally fast responses or filtering out anyone who picks the same answer several questions in a row. These methods work well against bots (or inattentive humans) that give random or rule-based inputs, but AI agents can now generate responses that are increasingly coherent and hard to distinguish from real panelists’. I heard the phrase “existential threat” thrown around multiple times during discussions of opt-in response quality.
David Dutwin of NORC’s AmeriSpeak panel gave an excellent overview of where experts have landed. Only half of survey researchers report using any fraud-detection tools, and one reputable panel provider shared that its in-house tools now block nearly 30% of would-be respondents. Even the best panels flag at least 20% of respondents as likely fraudulent. Dutwin shared others’ research suggesting metadata fields, like email names and IP addresses, as a promising path forward.
Andrew Timm and his colleagues at GrowProgress shared their own fraud-detection methods. He noted how quickly AI agents have learned to identify questions meant to be invisible or unanswerable to humans (for example, “How much of the Lorem Ipsum text do you remember?” should stump real people but in the past agents would attempt to answer it at length). These “honeypot” questions were cutting-edge a year ago and are already losing their effectiveness. Similarly, mainstream LLMs are no longer vulnerable to prompt injection. Currently, “biometric” checks like typing cadence or mouse movement are harder for agents to mimic, since their job is mostly reading text and reasoning…for now.
The bottom line is that anyone relying on nonprobability panels should be talking regularly with their research team about respondent quality and fraud detection. And every data provider, from panel vendors to polling firms, should be thinking about how a determined bot farm might already be working around last month’s defenses.