Classifying Political Orientation on Twitter: It’s Not Easy!

Abstract

Numerous papers have reported great success at inferring the political orientation of Twitter users. This paper has some unfortunate news to deliver: while past work has been sound and often methodologically novel, we have discovered that reported accuracies have been systemically overoptimistic due to the way in which validation datasets have been collected, reporting accuracy levels nearly 30% higher than can be expected in populations of general Twitter users. Using careful and novel data collection and annotation techniques, we collected three different sets of Twitter users, each characterizing a different degree of political engagement on Twitter - from politicians (highly politically vocal) to "normal" users (those who rarely discuss politics). Applying standard techniques for inferring political orientation, we show that methods which previously reported greater than 90% inference accuracy, actually achieve barely 65% accuracy on normal users. We also show that classifiers cannot be used to classify users outside the narrow range of political orientation on which they were trained. While a sobering finding, our results quantify and call attention to overlooked problems in the latent attribute inference literature that, no doubt, extend beyond political orientation inference: the way in which datasets are assembled and the transferability of classifiers.

Which Node Attribute Prediction Task Are We Solving? Within-Network, Across-Network, or Across-Layer Tasks

“…This work highlights an important distinction between across-network and across-layer problems. Our investigation into across-network prediction follows other applications studying networks across varied social settings such as comparing heterogeneity across different networks (Jacobs et al 2015), evaluating link prediction across different networks (Dong et al 2012), and demonstrating how prediction problems can be more difficult than previously appreciated (Cohen and Ruths 2013).…”

Organizations Are Users Too: Characterizing and Detecting the Presence of Organizations on Twitter

“…Distinguishing account types can be viewed as a type of latent attribute inference, which aims to infer various properties of online accounts. While only recently has latent attribute inference work begun to examine the organizationperson distinction, much work has been done on other specific aspects such as political affiliation (Cohen and Ruths 2013), gender (Ciot, Sonderegger, and Ruths 2013;Alowibdi, Buy, and Yu 2013), age (Nguyen, Smith, and Rosé 2011;Nguyen et al 2013), location (Jurgens 2013), or combinations thereof (Zamal, Liu, and Ruths 2012;Li, Ritter, and Hovy 2014). Our work is complementary and may offer an important benefit by removing noise from organizational accounts which do not have human attributes.…”

“…Their dataset consists of tweets from 58 organizations, gathered from Twellow, and 600 personal accounts, identified by matching the account's profile description with a list of person names. Given the diversity of organization types and bias inherent to Twellow (Cohen and Ruths 2013), 58 organizational accounts offer a very sparse (and quite likely biased) sampling. Second, Yin et al (2014) performed an analysis using a sample of 5000 accounts that posted at least 200 geotagged tweets.…”

Analyzing the Political Sentiment of Tweets in Farsi

“…Unfortunately, many of these methods have been applied to classifying politicians or news sources, as opposed to more ordinary users. Cohen and Ruths have demonstrated that these methods, which typically report accuracy as high as 90%, achieve only 65% accuracy for normal users (Cohen and Ruths 2013).…”