Automated Text Classification of News Articles: A Practical Guide

Abstract

Automated text analysis methods have made possible the classification of large corpora of text by measures such as topic and tone. Here, we provide a guide to help researchers navigate the consequential decisions they need to make before any measure can be produced from the text. We consider, both theoretically and empirically, the effects of such choices using as a running example efforts to measure the tone of New York Times coverage of the economy. We show that two reasonable approaches to corpus selection yield radically different corpora and we advocate for the use of keyword searches rather than predefined subject categories provided by news archives. We demonstrate the benefits of coding using article segments instead of sentences as units of analysis. We show that, given a fixed number of codings, it is better to increase the number of unique documents coded rather than the number of coders for each document. Finally, we find that supervised machine learning algorithms outperform dictionaries on a number of criteria. Overall, we intend this guide to serve as a reminder to analysts that thoughtfulness and human validation are key to text-as-data methods, particularly in an age when it is all too easy to computationally classify texts without attending to the methodological choices therein.

Four best practices for measuring news sentiment using ‘off-the-shelf’ dictionaries: a large-scale p-hacking experiment

“…These four best practices are hardly original: most of them have been proposed in previous best practice articles (e.g. Grimmer & Stewart, 2013;Barberá, Boydstun, Linn, McMahon, & Nagler, 2020;Van Atteveldt & Peng, 2018). With our empirical findings, this discussion illustrates the importance of these best practices.…”

“…For example, Fogel-Dror, Shenhav, Sheafer, and Van Atteveldt (2018) utilize off-the-shelf LSD in an analysis of sentiment against news entities using a validated, rule-based approach. If one has to hand annotate 1% of the material and that amounts to a few thousand articles, a new study shows that there is more than enough data to train and validate an accurate supervised machine learning model of news sentiment (Barberá et al, 2020). Regardless, all these new applications require heavy human validation.…”

Social Media and Political Agenda Setting

“…9. As a point of comparison, Barberá et al (2021) report an accuracy of 71% for the classification of sentiment on the level of sentences. 10.…”

Computational Social Science and the Study of Political Communication

“…This very famously includes data documenting user behavior in digital environments -so-called digital trace data (Freelon, 2014;Golder & Macy, 2014;Howison et al, 2011;Jungherr, 2015). Increasingly, however, other large datasets that are relevant for political communication research have become available digitally, such as large text corpora covering fields as diverse as newspaper coverage (Barberá et al, 2020), literature (Piper, 2018;Underwood, 2019), historical or contemporary parliamentary speeches (Rauh & Schwalbach, 2020), and images (Williams et al, 2020). All these datasets are legitimate objects of computational communication, and it is thus unnecessarily limiting to restrict one's definition of CSS to specific types of datasets.…”