The False Positive Problem of Automatic Bot Detection in Social Science Research

Rauchfleisch, Adrian; Kaiser, Jonas

doi:10.2139/ssrn.3565233

Cited by 42 publications

(13 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Examples abound. Activity on social media has been used to label users according to their political ideology (Barberá, 2015), psychological traits (Azucar et al, 2018), mental health (Chancellor & Choudhury, 2020), or the authenticity of their account (Rauchfleisch & Kaiser, 2020). Labels are powerful tools in CSS as they allow large-scale automated assignment of interventions based on perceived traits or preferences of users.…”

Section: Labels: Can We Infer People's Traits Based On Digital Traces?mentioning

confidence: 99%

“…Increasingly, early enthusiasm is replaced by skepticism. Careful examination shows that methods labeling accounts as bots have not proved to be reliable, with mislabeling of actual authentic and legitimate accounts as bots (false-positive) and strong temporal decay in precision (out of sample prediction) (Rauchfleisch & Kaiser, 2020).…”

Section: Labels: Can We Infer People's Traits Based On Digital Traces?mentioning

confidence: 99%

See 1 more Smart Citation

Computational Social Science and the Study of Political Communication

Theocharis

Jungherr

2020

Political Communication

View full text Add to dashboard Cite

The challenge of disentangling political communication processes and their effects has grown with the complexity of the new political information environment. But so have scientists' toolsets and capacities to better study and understand them. We map the challenges and opportunities of developing, synthesizing, and applying data collection and analysis techniques relying primarily on computational methods and tools to answer substantive theory-driven questions in the field of political communication. We foreground the theoretical, empirical, and institutional opportunities and challenges of Computational Communication Science (CCS) that are relevant to the political communication community. We also assess understandings of CCS and highlight challenges associated with data and resource requirements, as well as those connected with the theory and semantics of digital signals. With an eye to existing practices, we elaborate on the key role of infrastructures, academic institutions, ethics, and training in computational methods. Finally, we present the six full articles and two forum contributions of this special issue illustrating methodological innovation, as well as the theoretical, practical, and institutional relevance and challenges for realizing the potential of computational methods in political communication..

show abstract

Section: Labels: Can We Infer People's Traits Based On Digital Traces?mentioning

confidence: 99%

Section: Labels: Can We Infer People's Traits Based On Digital Traces?mentioning

confidence: 99%

Computational Social Science and the Study of Political Communication

Theocharis

Jungherr

2020

Political Communication

View full text Add to dashboard Cite

show abstract

“…A further problem is that many genuine Twitter users deploy automated or semiautomated software to help manage their tweeting and followership patterns, which can make them appear bot-like. Notwithstanding these limitations, botometer is a scalable and accurate way of identifying bots and represents state-of-the-art for this task (Rauchfleisch and Kaiser 2020).…”

Section: The Demographics Of Rt's Twitter Audiencementioning

confidence: 99%

Understanding RT’s Audiences: Exposure Not Endorsement for Twitter Followers of Russian State-Sponsored Media

Crilley

Gillespie

Vidgen

et al. 2020

The International Journal of Press/Politics

View full text Add to dashboard Cite

The Russian state-funded international broadcaster RT (formerly Russia Today) has attracted much attention as a purveyor of Russian propaganda. To date, most studies of RT have focused on its broadcast, website, and social media content, with little research on its audiences. Through a data-driven application of network science and other computational methods, we address this gap to provide insight into the demographics and interests of RT’s Twitter followers, as well as how they engage with RT. Building upon recent studies of Russian state-sponsored media, we report three main results. First, we find that most of RT’s Twitter followers only very rarely engage with its content and tend to be exposed to RT’s content alongside other mainstream news channels. This indicates that RT is not a central part of their online news media environment. Second, using probabilistic computational methods, we show that followers of RT are slightly more likely to be older and male than average Twitter users, and they are far more likely to be bots. Third, we identify thirty-five distinct audience segments, which vary in terms of their nationality, languages, and interests. This audience segmentation reveals the considerable heterogeneity of RT’s Twitter followers. Accordingly, we conclude that generalizations about RT’s audience based on analyses of RT’s media content, or on vocal minorities among its wider audiences, are unhelpful and limit our understanding of RT and its appeal to international audiences.

show abstract

“…In response, many researchers have proposed a range of detection mechanisms for identifying bots, trolls, and botnets (e.g., some examples include [1,29,30,33]). While the bot detection has been widely studied for the past decade [6,15,22], the evolving nature of social media sites has made bot detection less promising due to their focus on identifying automation, lack of providing intent, and only looking at singular accounts. Bot detectors are not applicable for detecting the inauthentic coordinated behavior that SIO campaigns show as these use multiple accounts to push their rhetoric at scale.…”

Section: Introductionmentioning

confidence: 99%

On the Detection of Disinformation Campaign Activity with Network Analysis

Vargas

Emami

Traynor

2020

Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop

View full text Add to dashboard Cite

Online manipulation of information has become more prevalent in recent years as state-sponsored disinformation campaigns seek to influence and polarize political topics through massive coordinated efforts. In the process, these efforts leave behind artifacts, which researchers have leveraged to analyze the tactics employed by disinformation campaigns after they are taken down. Coordination network analysis has proven helpful for learning about how disinformation campaigns operate; however, the usefulness of these forensic tools as a detection mechanism is still an open question. In this paper, we explore the use of coordination network analysis to generate features for distinguishing the activity of a disinformation campaign from legitimate Twitter activity. Doing so would provide more evidence to human analysts as they consider takedowns. We create a time series of daily coordination networks for both Twitter disinformation campaigns and legitimate Twitter communities, and train a binary classifier based on statistical features extracted from these networks. Our results show that the classifier can predict future coordinated activity of known disinformation campaigns with high accuracy (1 = 0.98). On the more challenging task of out-of-distribution activity classification, the performance drops yet is still promising (1 = 0.71), mainly due to an increase in the false positive rate. By doing this analysis, we show that while coordination patterns could be useful for providing evidence of disinformation activity, further investigation is needed to improve upon this method before deployment at scale.

show abstract

The False Positive Problem of Automatic Bot Detection in Social Science Research

Cited by 42 publications

References 23 publications

Computational Social Science and the Study of Political Communication

Computational Social Science and the Study of Political Communication

Understanding RT’s Audiences: Exposure Not Endorsement for Twitter Followers of Russian State-Sponsored Media

On the Detection of Disinformation Campaign Activity with Network Analysis

Contact Info

Product

Resources

About