Bot, or not? Comparing three methods for detecting social bots in five political discourses

Abstract

Social bots – partially or fully automated accounts on social media platforms – have not only been widely discussed, but have also entered political, media and research agendas. However, bot detection is not an exact science. Quantitative estimates of bot prevalence vary considerably and comparative research is rare. We show that findings on the prevalence and activity of bots on Twitter depend strongly on the methods used to identify automated accounts. We search for bots in political discourses on Twitter, using three different bot detection methods: Botometer, Tweetbotornot and “heavy automation”. We drew a sample of 122,884 unique user Twitter accounts that had produced 263,821 tweets contributing to five political discourses in five Western democracies. While all three bot detection methods classified accounts as bots in all our cases, the comparison shows that the three approaches produce very different results. We discuss why neither manual validation nor triangulation resolves the basic problems, and conclude that social scientists studying the influence of social bots on (political) communication and discourse dynamics should be careful with easy-to-use methods, and consider interdisciplinary research.

The digital spectator activities of Twitter users on the issue of the anti-terrorism bill in the Philippines

“…These were considered to automatically refrain discussions from possible fraudulent local troll accounts or Twitter bots that pose unreasonable behavior on Twitter. This is a measure to avoid newly created bot and burn accounts that only resurface on Twitter when an issue is popular or trending on the platform (Martini et al, 2021).…”

Botometer 101: social bot practicum for computational social scientists

“…The accuracy of the model may further decay when dealing with new accounts different from those in the training datasets. These accounts might come from a different context, use different languages other than English [ 52 , 53 ], or show novel behavioral patterns [ 34 , 45 , 54 ]. These limitations are inevitable for all supervised machine learning algorithms, and are the reasons why Botometer has to be upgraded routinely.…”

Botometer 101: Social bot practicum for computational social scientists

“…The accuracy of the model may further decay when dealing with new accounts different from those in the training datasets. These accounts might come from a different context, use different languages other than English [48,49], or show novel behavioral patterns [43,34,50]. These limitations are inevitable for all supervised machine learning algorithms, and are the reasons why Botometer has to be upgraded routinely.…”