An Examination of the Use of Large Language Models to Aid Analysis of Textual Data

Abstract

Increasing use of machine learning and Large Language Models (LLMs) opens up opportunities to use these artificially intelligent algorithms in novel ways. In this article we propose a methodology using LLMs to support traditional deductive coding in qualitative research. We began our analysis with three different sample texts taken from existing interviews. Next, we created a codebook and inputted the sample text and codebook into an LLM. We asked the LLM to determine if the codes were present in a sample text provided and requested evidence to support the coding. The sample texts were inputted 160 times to record changes between iterations of the LLM response. Each iteration was analogous to a new coder deductively analyzing the given text with the codebook information. In our results, we present the outputs for these recursive analyses, along with a comparison of the LLM coding to evaluations made by human coders using traditional coding methods. We argue that LLM analysis can aid qualitative researchers by deductively coding transcripts, by providing a systematic and reliable platform for code identification, and by offering a means of avoiding analysis misalignment. Implications of using LLM in research praxis are discussed, along with current limitations.

Exploring the Potential and Challenges of WhatsApp Focus Groups in Qualitative Consumer Research: A Case Study of Sustainable Eating

“…The LLM has been found to improve the objectivity of human reasoning in topic coding and content analysis when studying qualitative data (Bano et al, 2024). In the focus group study, LLM was used alongside NVivo to validate the manually coded topics by respondents' answers with basic prompting to contextualise the questions in the analysis (Tai et al, 2023 ). This provided an additional layer of quality assurance to guarantee precise coding procedures that are typically conducted by comparing the discovered topics between researchers.…”