Can Generative AI improve social science?

Abstract

Generative AI that can produce realistic text, images, and other human-like outputs is currently transforming many different industries. Yet it is not yet known how such tools might influence social science research. I argue Generative AI has the potential to improve survey research, online experiments, automated content analyses, agent-based models, and other techniques commonly used to study human behavior. In the second section of this article, I discuss the many limitations of Generative. I examine how bias in the data used to train these tools can negatively impact social science research—as well as a range of other challenges related to ethics, replication, environmental impact, and the proliferation of low-quality research. I conclude by arguing that social scientists can address many of these limitations by creating open-source infrastructure for research on human behavior. Such infrastructure is not only necessary to ensure broad access to high-quality research tools, I argue, but also because the progress of AI will require deeper understanding of the social forces that guide human behavior.

Perils and opportunities in using large language models in psychological research

“…Each time modifications are made to ChatGPT’s algorithms for performance enhancement, the nature and scope of biases embedded in the model may change. This could impact the effectiveness of previously established “best practices” and debiasing strategies ( 15 , 100 ). Currently, ChatGPT, and to our best knowledge any other closed-source model, does not freely provide past versions that allow researchers to use the model from specific points in time (e.g.…”

“…After each major update, it only provides a snapshot, which is then deprecated within 3 months to 1 year. This means that even when updates address and mitigate detected biases, they also introduce the potential for “process reproducibility failure” ( 98 ) in the generated data and impede reproducibility, critical for scientific rigor ( 15 , 100 , 101 ). This also affects currently emerging practices for reproducibility, such as researchers sharing prompts and Application Programming Interface (API) parameters because the effect of these parameters (or the parameters themselves) and the outputs generated from these prompts can change over time.…”

“…Researchers should be aware of the current black-box nature of LLMs ( 15 , 100 ) not only from an open-science standpoint. More generally, researchers should be interested in access to high-quality, informative semantic representations and the algorithms used to generate outputs, instead of only the outputs.…”

“…Additionally, these models’ capabilities in text analysis and generation could possibly be harnessed by researchers as an easy-to-use method for analyzing textual data, such as coding texts for mental health assessment ( 14 ). There has recently been a rapid string of psychological research output related to and facilitated by these models ( 13 , 15–17 ). In particular, ChatGPT has been employed in a range of social and behavioral applications, from hate-speech classification ( 18 ) to sentiment analysis ( 19 ), often with promising results.…”