Does GPT-3 know what the Most Important Issue is? Using Large Language Models to Code Open-Text Social Survey Responses At Scale

Applying machine-learning to rapidly analyze large qualitative text datasets to inform the COVID-19 pandemic response: comparing human and machine-assisted topic analysis techniques

“…In the rapidly evolving landscape of artificial intelligence, the emergence and application of Large Language Models (LLMs) like Chat-GPT in qualitative data analysis needs to be considered as a viable alternative approach. Mellon et al (34) demonstrated how LLMs can accurately replicate human coding of large-scale data when classifying the most important perceived issues in the United Kingdom, such as health and education. However, whilst LLMs offer scalability and efficiency, it is possible that they could inadvertently introduce biases or miss nuanced interpretations if applied uncritically (35), and researchers must ensure they manually validate the output to verify accuracy and quality.…”

“…However, whilst LLMs offer scalability and efficiency, it is possible that they could inadvertently introduce biases or miss nuanced interpretations if applied uncritically (35), and researchers must ensure they manually validate the output to verify accuracy and quality. Furthermore, Mellon et al (34) highlight that it is currently unknown whether LLMs can code the sentiment of open-text data, or whether they are capable of coding the data as well as producing a coding scheme in the way that STM does. It is possible that current and future iterations of Chat-GPT could have these capabilities, but further research is required.…”

Crowdsourced Adaptive Surveys

“…Despite the advantages of the CSAS method, scholars should be aware of potential limitations or drawbacks of using LLMs. Simpler classification tasks, like those demonstrated in the work on important issue identification (Mellon et al, 2022), may not require the additional investment of fine-tuning pre-trained LLMs, as their existing capabilities may suffice. However, capturing more abstract concepts such as misinformation may require additional steps such as fine-tuning and few-shot learning (i.e., including positive and negative examples in an LLM prompt).…”

“…The question bank was seeded with eight popular issue areas taken from Gallup. In this study, a more capable LLM, OpenAI's GPT-4, was used to convert unstructured text into issue topics and filter out redundant issues in a single operation.6 Recent studies have shown that in the task of classifying open-ended text to identify "most important issues," the efficacy of Large Language Models (LLMs) like GPT-4 is on par with classification algorithms trained on thousands of examples, achieving performance levels marginally below that of human evaluators(Mellon et al, 2022). User-submitted items spanned salient issues such as abortion and climate change, and less salient issues such as universal basic income and deficit spending.…”

ChatGPT for (Finance) Research: The Bananarama Conjecture

“…2 Chen and Eger (2022) also explore use in title and abstract generation, and in the domain of finance, Wenzlaff and Spaeth (2022) are able to generate reasonably academically-appropriate definitions of new financial concepts. Mellon et al (2022) explores one aspect of the application to research testing, by showing the platform can be useful as a complement to scoring open-text survey results. While Adesso (2022) has used GPT3 to write a full paper in physics, to be submitted to a journal ''as is '', and Zhai (2022) has also experimented with creating a research paper outline.…”