The Augmented Social Scientist. Using Sequential Transfer Learning to Annotate Millions of Texts with Human-Level Accuracy

Abstract

The last decade witnessed a spectacular rise in the volume of available textual data. With this new abundance came the question of how to analyze it. In the social sciences, scholars mostly resorted to two well-established approaches, human annotation on sampled data on the one hand (either performed by the researcher, or outsourced to microworkers), and quantitative methods on the other. Each approach has its own merits - a potentially very fine-grained analysis for the former, a very scalable one for the latter - but the combination of these two properties has not yielded highly accurate results so far. Leveraging recent advances in sequential transfer learning, we demonstrate via an experiment that an expert can train a precise, efficient automatic classifier in a limited amount of time. We also show that, under certain conditions, researchers can also outsource the annotation to other groups. We demonstrate these points using a classic research question in the sociology of journalism, the rise of “horse race” coverage of politics. We conclude that recent advances in transfer learning help us augment ourselves when analyzing unstructured data.

A Systematic Evaluation of Text Mining Methods for Short Texts: Mapping Individuals’ Internal States from Online Posts

“…We find that advanced supervised machine learning classification methods using transformer language models can approach the performance of human analysis when it comes to inference on various internal states from short texts. Our results, thus, echo recent suggestions about the potential of deep learning methods in social science applications (van Atteveldt et al 2021;Bonikowski, Luo, and Stuhler 2022;Do, Ollion, and Shen 2022;Widmann and Wich 2022). Yet, we also suggest that increased method complexity does not always warrant a large improvement in performancesimple supervised machine learning methods such as logistic regression can sometimes perform almost as well as more complex algorithms.…”

“…Existing evidence suggests that crowdsourcing works rather well for simpler coding tasks (Benoit et al 2016), while more complex 4 Corpus construction also calls for important decisions which are beyond our scope here (yet, see Bonikowski and Nelson 2022). Depending on the application, researchers might choose to work with whole texts, sentences, or text segments as the unit of analysis (for latter, see approaches in Barberá et al 2021 andDo, Ollion, andShen 2022).…”

“…First, we build a 100-dimensional word2vec ) model using our manually coded data. 13 Second, we use several powerful transformer models in which the word vectors were built by training the models on large amounts of textual data to capture word usages across various contexts (Do et al 2022;): (1) BERT, pre-trained on the BookCorpus of 800 million words and the corpus of English Wikipedia of 2,500 million words );…”

Seeded Topic Models in Digital Archives: Analyzing the Swedish Understanding of Immigration, 1945--2019

“…For many in the social sciences, computational text analysis comes in two variants, as either supervised and unsupervised methods. Supervised methods rest on the researcher's access to labels for meaning structures in text data, such as categories and a coding scheme, and extrapolate these labels on unseen text (Nelson et al 2021, Chen et al 2018, Lichtenstein & Rucks-Ahidiana 2021, Do et al 2022. Unsupervised methods, by contrast, infer information about language patterns, such as co-occurrences of words in documents, without drawing on predefined categories or coding schemes.…”

Curating Training Data for Reliable Large-Scale Visual Data Analysis: Lessons from Identifying Trash in Street View Imagery

“…They offer a way to draw on large numbers of people to gather information or complete a large task, such as developing a large dataset to train an ML model. A growing number of studies within the social sciences have used crowdsourcing to develop training data for ML processes, but they have primarily been used to analyze large-scale text corpuses (e.g., Benoit et al 2016; Benoit, Munger, and Spirling 2019; Budak, Goel, and Rao 2016; Do, Ollion, and Shen 2022; Nelson et al 2021; Wilkerson and Casas 2017; Ying, Montgomery, and Stewart 2022). Scholars in computer science have taken this approach with visual data to examine perceptions of streetscapes (Naik et al 2014) and the make and model of cars in images (Gebru et al 2017).…”