Tradeoffs in Accuracy and Efficiency in Supervised Learning Methods

Abstract

Text is becoming a central source of data for social science research. With advances in digitization and open records practices, the central challenge has in large part shifted away from availability to usability. Automated text classification methodologies are becoming increasingly important within political science because they hold the promise of substantially reducing the costs of converting text to data for a variety of tasks. In this paper, we consider a number of questions of interest to prospective users of supervised learning methods, which are appropriate to classification tasks where known categories are applied. For the right task, supervised learning methods can dramatically lower the costs associated with labeling large volumes of textual data while maintaining high reliability and accuracy. Information science researchers devote considerable attention to comparing the performance of supervised learning algorithms and different feature representations, but the questions posed are often less directly relevant to the practical concerns of social science researchers. The first question prospective social science users are likely to ask is -how well do such methods work? The second is likely to be -how much do they cost in terms of human labeling effort? Relatedly, how much do marginal improvements in performance cost? We address these questions in the context of a particular dataset -the Congressional Bills Project -which includes more than 400,000 labeled bill titles (19 policy topics). This corpus also provides opportunities to experiment with varying sample sizes and sampling methodologies. We are ultimately able to locate an accuracy/efficiency sweet spot of sorts for this dataset by leveraging results generated by an ensemble of supervised learning algorithms.

Development and testing of a text-mining approach to analyse patients’ comments on their experiences of colorectal cancer care

“…The quality of the algorithm results depend on three factors 37. First, the quality of the data provided may make it more ‘difficult’ or ‘easy’ it is to identify patterns.…”

The influence of temperature on #ClimateChange and #GlobalWarming discourses on Twitter

“…Specifically, ForSight has been verified through comparison with surveys data and election results [Ceron et al, 2014;Hitlin, 2015]. These scholars, among others, have also verified the resilience of supervised learning programs based on the training set used for the program [Collingwood and Wilkerson, 2012;Hopkins and King, 2010]. Using a large and randomly distributed subset of the sample posts improves the accuracy of the program, in addition to extensive human coding [Collingwood and Wilkerson, 2012;Neuendorf, 2017].…”

“…Scholars have argued for applying this hybrid content analysis method to social media discourses as it possesses the reliability and efficiency of computer-based coding while preserving the latent validity of human coding Su, Akin and Brossard, 2017]. Others have examined and verified such supervised learning programs [Collingwood and Wilkerson, 2012]. Specifically, ForSight has been verified through comparison with surveys data and election results [Ceron et al, 2014;Hitlin, 2015].…”

Clause Analysis: Using Syntactic Information to Automatically Extract Source, Subject, and Predicate from Texts with an Application to the 2008–2009 Gaza War

“…Such approaches have been successfully used for automatically determining the topic of a text, for example using machine learning (Sebastiani, 2002). In the comparative agendas project, a technique known as active learning was used to accurately determine the topic of large amounts of documents at a fraction of the cost of coding all documents (Hillard et al, 2008;Collingwood and Wilkerson, 2012). Recent work on unsupervised topic models (i.e.…”