Do Machines Replicate Humans? Toward a Unified Understanding of Radicalizing Content on the Open Social Web

Abstract

The advent of the Internet inadvertently augmented the functioning and success of violent extremist organizations. Terrorist organizations like the Islamic State in Iraq and Syria (ISIS) use the Internet to project their message to a global audience. The majority of research and practice on web‐based terrorist propaganda uses human coders to classify content, raising serious concerns such as burnout, mental stress, and reliability of the coded data. More recently, technology platforms and researchers have started to examine the online content using automated classification procedures. However, there are questions about the robustness of automated procedures, given insufficient research comparing and contextualizing the difference between human and machine coding. This article compares output of three text analytics packages with that of human coders on a sample of one hundred nonindexed web pages associated with ISIS. We find that prevalent topics (e.g., holy war) are accurately detected by the three packages whereas nuanced concepts (Lone Wolf attacks) are generally missed. Our findings suggest that naïve approaches of standard applications do not approximate human understanding, and therefore consumption, of radicalizing content. Before radicalizing content can be automatically detected, we need a closer approximation to human understanding.

A survey on extremism analysis using natural language processing: definitions, literature review, trends and challenges

“…( 2020 ); Hall et al. ( 2020 ); Owoeye and Weir ( 2018 ); Macnair and Frank ( 2018 ); Figea et al. ( 2016 ); Scrivens and Frank ( 2016 ); Scrivens et al.…”

Countering Extremists on Social Media: Challenges for Strategic Communication and Content Moderation

“…Given the high risks of incorrect flags that lead to takedown of innocent users and their content, auditing and evaluating AI approaches at use in content moderation is of significant concern, especially considering the demonstrable biases against women and minorities that studies of algorithms have revealed (Eubanks, 2018; Noble, 2018). While many projects have focused on how to detect extremist content, Hall, Logan, Ligon, and Derrick (2020) instead evaluate the performance of machines against human judgement, probing the limits of text‐based methods for the classification of extremism. They find that for jihadist content, approaches to detect extremist content with AI require significant work in integrating human understanding into machine abilities.…”

“…While these approaches perform well for high‐level concepts, humans provide more granular analysis that identifies key themes and forms of content, such as emotion. By engaging in a validation of open‐source AI tools in detection of extremist content, Hall et al (2020) provide valuable advancements in research design and methodology that can be applied for future study; probing the possibilities and limits of technical systems in primary CVE, and identifying key challenges that software must surmount for it to be a viable alternative to human‐led moderation.…”

“…As the authors of the final contribution to this special issue note, alternative media on the political right “results in a combination of an anti‐hegemonic impetus” and a wide range of both mainstream and extreme political positions, ranging from economic liberalism to nativism (Hall et al, 2020). While research on far‐right exploitation of social media is increasing, much of this work focuses on representation, narratives, ideology, and discourse (Deem, 2019; Froio & Ganesh, 2019; Klein & Muis, 2019; Richards, 2019; Topinka, 2018), as well as disinformation spreading from right‐wing digital news to social media platforms and mainstream media (Benkler et al, 2018; Bennett & Livingston, 2018; Marwick & Lewis, 2017).…”

Assessment Approach to the Interconnections of Social Network Radical Online Communities Based on Open Data Analysis with the Use of TF-IDF

“…Ligon and D.C. Derrick, authors of the work "Do Machines Replicate Humans? Toward a Unified Understanding of Radicalizing Content on the Open Social Web», the easiness of destructive content spreading, as well as a variety of its delivery methods provide radical communities with access to a massive audience, which allows solving the problems of searching and recruiting supporters, spreading ideologically significant information, attracting resources and providing general management [1]. Herewith, the Internet itself is not the cause of radicalization, but only a toolkit platform that provides previously unreachable massiveness and speed of social communications.…”