Large language models as a substitute for human experts in annotating political text

Abstract

Large-scale text analysis has grown rapidly as a method in political science and beyond. To date, text-as-data methods rely on large volumes of human-annotated training examples, which place a premium on researcher resources. However, advances in large language models (LLMs) may make automated annotation increasingly viable. This paper tests the performance of GPT-4 across a range of scenarios relevant for analysis of political text. We compare GPT-4 coding with human expert coding of tweets and news articles across four variables (whether text is political, its negativity, its sentiment, and its ideology) and across four countries (the United States, Chile, Germany, and Italy). GPT-4 coding is highly accurate, especially for shorter texts such as tweets, correctly classifying texts up to 95% of the time. Performance drops for longer news articles, and very slightly for non-English text. We introduce a ‘hybrid’ coding approach, in which disagreements of multiple GPT-4 runs are adjudicated by a human expert, which boosts accuracy. Finally, we explore downstream effects, finding that transformer models trained on hand-coded or GPT-4-coded data yield almost identical outcomes. Our results suggest that LLM-assisted coding is a viable and cost-efficient approach, although consideration should be given to task complexity.

Accuracy of a large language model in distinguishing anti- and pro-vaccination messages on social media: The case of human papillomavirus vaccination

“…The language model also encountered difficulties in accurately replicating human evaluation decisions for neutral messages. Additionally, the model’s accuracy was lower for short-form messages than long-form ones, differing from findings in a study on political texts ( Heseltine and Clemm von Hohenberg, 2024 ).…”

“…LLMs have demonstrated considerable capacity for human-level decision-making and logical processing ( Katz et al, 2023 , Liu et al, 2023 ). Furthermore, the increasing accessibility and user-friendliness of these powerful LLMs are amplifying their impacts in various academic disciplines ( Ziems et al, 2023 , Heseltine and Clemm von Hohenberg, 2024 ).…”