ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification

Abstract

ChatGPT has shown strong capabilities in natural language generation tasks, which naturally leads researchers to explore where its abilities end. In this paper, we examine whether ChatGPT can be used for zero-shot text classification, more specifically, automatic genre identification. We compare ChatGPT with a multilingual XLM-RoBERTa language model that was fine-tuned on datasets, manually annotated with genres. The models are compared on test sets in two languages: English and Slovenian. Results show that Chat-GPT outperforms the fine-tuned model when applied to the dataset which was not seen before by either of the models. Even when applied on Slovenian language as an underresourced language, ChatGPT's performance is no worse than when applied to English. However, if the model is fully prompted in Slovenian, the performance drops significantly, showing the current limitations of ChatGPT usage on smaller languages. The presented results lead us to questioning whether this is the beginning of an end of laborious manual annotation campaigns even for smaller languages, such as Slovenian.

A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

“…The evaluation is done in zero and few-shot settings using different prompting strategies like chainof-thought (CoT) [125], [127], [134], [137], [138], [141], [144], self-question prompting (SQP) [138], clue and reasoning prompting (CARP) [144] etc. Most of the research works focused on English datasets, except a few research works focused on other languages like Chinese [128], Slovenian [131], Indonesian [132], Javanese [132], and Buginese [132]. A brief summary of research works exploring GLLMs for various text classification problems is presented in Table 1.…”

“…Some of the research works demonstrated that GPT-3 family large language models can outperform taskspecific fine-tuned models [131], [134] and domainspecific LLMs [136]. Kuzman et al [131] showed that ChatGPT outperforms fine-tuned XLM-R model in the task of automatic genre identification in the English language.…”

ChatGPT: A Brief Narrative Review

“…Moreover, it delivers no sources or footnotes regarding where to discover the content. Therefore, it is not perfect to implement this bot by itself for digital tracking and study [38].…”

Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark

“…ChatGPT raises hopes to take over this labor, cost, and time intensive work by either annotating text as a basis for training data in further analyses (e.g., using established procedures that are cheaper and faster) or by directly classifying texts. Several studies already discuss and demonstrate promising zero-shot applications of ChatGPT (i.e., classification of unseen data without any training on the classification task), for example for detecting hate speech, misinformation, rating the credibility of news outlets, and more (Hoes et al, 2023;Huang et al, 2023;Kuzman et al, 2023;Qin et al, 2023;Yang & Menczer, 2023;Zhong et al, 2023).…”

“…What has been investigated less so far is the consistency of ChatGPT's zero-shot text classification and text annotation capabilities. While this is not necessarily an important issue when robust validation takes place (e.g., against human-annotated data), it can be an issue if ChatGPT comes to be seen as the end of manual data annotation (Kuzman et al, 2023) and validation efforts are 3 neglected. The inclusion of ChatGPT in quantitative and qualitative text annotation and analysis software that offers "full-automatic data coding" (Atlas.ti, 2023) justifies the concerns of unvalidated text annotation.…”