Demonstrations of the Potential of AI-based Political Issue Polling

Abstract

Political polling is a multi-billion-dollar industry with outsized influence on the societal trajectory of the United States and nations around the world. However, in recent years it has been severely challenged by rising nonresponse rates and other factors that stress its cost, availability, and accuracy. At the same time, artificial intelligence (AI) chatbots such as ChatGPT have become highly compelling stand-ins for a wide range of human behavior, powered by increasingly sophisticated large language models (LLMs). Because these LLMs are trained on huge corpora of writing by diverse people captured from across the Internet, they are potentially capable of representing a wide range of beliefs on many policy issues. Could AI chatbots be an effective tool for anticipating public opinion on controversial issues to the extent that they could be used by campaigns, interest groups, and polling firms?We have developed a prompt engineering methodology for eliciting humanlike survey responses from ChatGPT, which simulate the response to a policy question of a person described by a set of demographic and ideological factors, and produce both an ordinal numeric response score and a textual justification. We execute large-scale experiments using this method, querying GPT for thousands of simulated responses at a cost that is at least three orders of magnitude lower than human surveys. We compare this simulated data to human issue polling data from the Cooperative Election Study (CES).We find that ChatGPT is sometimes effective at anticipating both the mean level and distribution (correlation , normalized Earth mover's distance ) of public opinion on a variety of policy issues well established in its training data such as abortion bans and approval of the US Supreme Court, particularly in their breakdown along ideological lines. However, it is much less successful at anticipating demographic (age, race, and gender) differences between respondents. Moreover, ChatGPT tends to overgeneralize its conception of ideological differences to new policy issues that arose after its training data was collected, such as American support for involvement in the war in Ukraine. Our work has implications for our understanding of the strengths and limitations of the current generation of AI chatbots as virtual publics or online listening platforms, future directions for LLM development, and applications of AI tools to the political domain.

Human Intelligence, Artificial Intelligence, and Homo sapiens Intelligence?

“…Yet, as Sanders et al (2023) indicate, we're merely at the dawn of understanding these interactions. In his interview, Lo exercised the same caution: "At this point, when it comes to financial advice, I would be extremely cautious about trusting advice from ChatGPT" (Lo et al, 2023).…”

“…Few of us would feel comfortable to have our high-stake-such as financial and medical-decisions be made by a person or an algorithm that we do not have some degree of trust in (granted, it could be a blind trust). The nuanced findings by Sanders et al (2023) about the reliability of using chatbots to gather political opinions remind us that the issue of trustworthy AI and data science should always be at the core of our pursuit of enhanced human intelligence and capability.…”

“…Together, the four articles highlighted in this section (Potok, 2023;Ruberg et al, 2023;Sanders et al, 2023;and Weeden, 2023) provide a snapshot of how academia, government, and industry make sector-wise as well as collective efforts to ensure the trustworthiness of data, data science, and ultimately the findings and policies derived from them.…”

Linking artificial intelligence facilitated academic misconduct to existing prevention frameworks

“…Significantly more detailed discussions of how transformer models work and both their strengths and weaknesses (including legality of content use, e.g., McKendrick 2022) can be found elsewhere (e.g., Dwivedi et al 2023), but beyond these technical discussions there has also been considerable debate around their potential societal implications. To illustrate, 'human forecasters' have labelled ChatGPT as the first step in the industrialisation of AI (Lowrey 2023), and LLMs as tools which might destabilise democracies (Sanders & Schneier 2023), or even break capitalism (Bove 2023).…”

2024: A Year of Crises, Change, Contemplation, and Commemoration

“…The year 2024 will mark another U.S. presidential election. Considering the public's eroded trust in polling during the 2016 and 2020 elections, 1 it's hardly a bold prediction to say that political pollsters will face yet another crisis in regaining public confidence (see ; as well as Sanders et al, 2023). More alarmingly, a far greater crisis is brewing.…”