Computer-Assisted Topic Classification for Mixed-Methods Social Science Research

Abstract

Social scientists interested in mixed-methods research have traditionally turned to human annotators to classify the documents or events used in their analyses. The rapid growth of digitized government documents in recent years presents new opportunities for research but also new challenges. With more and more data coming online, relying on human annotators becomes prohibitively expensive for many tasks. For researchers interested in saving time and money while maintaining confidence in their results, we show how a particular supervised learning system can provide estimates of the class of each document (or event). This system maintains high classification accuracy and provides accurate estimates of document proportions, while achieving reliability levels associated with human efforts. We estimate that it lowers the costs of classifying large numbers of complex documents by 80% or more.

Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts

“…Heuristically, as long as the classifiers are accurate and diverse, combining the classifiers will improve accuracy (Jurafsky and Martin 2009). But ensembles are also useful for other reasons, including: increased out-of-sample stability and the ability to capture complex functional forms with relatively simple classifiers (Dietterich 2000;Hillard, Purpura, and Wilkerson 2008). Schemes for developing ensembles are diverse.…”

“…In fact, a recent debate in political science casts unsupervised and supervised as competitor methods (e.g., Hillard, Purpura, and Wilkerson 2008;Quinn et al 2010). This debate is misplaced: supervised and unsupervised methods are different models with different objectives.…”

Analyzing Positions and Topics in Political Discussions of the German Bundestag

“…There are two general approaches to classify the topics of text: either the topics are known in advance and constitute a static set of categories, for example (Hillard et al, 2008), or they are unknown in advance and dynamically created depending on the data, as in (Quinn et al, 2010) (see also (Grimmer and Stewart, 2013) and (Sebastiani, 2002) for an overview). In our scenario, we assume a common set of topics over several data sources, namely the party manifestos and transcripts of speeches in our case.…”

Computer-Assisted Reading and Discovery for Student Generated Text in Massive Open Online Courses

“…The variety of benefits to computer-assisted text analysis over hand coding include the natural improvements in speed, the ability to process high volumes of text, and the consistency of treatment of all parts of the corpus (Grimmer & King, 2011;Hillard, Purpura, & Wilkerson, 2008;Lowe & Benoit, 2013). Humans often struggle with the development of complicated coding schemes (Quinn, Monroe, Colaresi, Crespin, & Radev, 2010), and there is some experimental evidence to suggest that humans judge clusters produced by automated methods to be more semantically coherent than even an taxonomy created by the documents' authors (Grimmer & King, 2011;Grimmer & Stewart, 2013).…”