Co‐Producing Security: Platform Content Moderation and European Security Integration

Abstract

The European Union (EU) seeks to play a leading role in steering the private work of online content moderation, as demonstrated by numerous policy and legislative initiatives in the domain. Two initiatives, in particular, are shaping terrorist content moderation: the creation of a EU Internet Referral Unit and the adoption of a Regulation on preventing the online dissemination of terrorist content (TERREG). This article analyses these initiatives and their practical effects. In particular, it unpacks the legal and technological mechanisms at the core of EU regulation in the realm of online terrorist content moderation, and how they co‐produce security decisions across public and private spheres. Based on interviews, fieldwork observations and document analysis, we show how processes of referral and removal, and processes of flagging and filtering are key to EU‐directed content moderation. In conclusion, we reflect on content moderation as a novel form of European security integration.

The Sanitised Platform

“…Major platforms often permit queer visibility only where it is desexualised, unthreatening and integrated into heteronormative family structures and values 92 . In the context of terrorist content -a regulatory priority for the EU -moderation unfolds through close cooperation between platforms and European security agencies 93 , which primarily target Islamist terrorism and have long histories of racist and Islamophobic discrimination 94 . Bloch-Wehba has shown how the way platforms define and identify terrorist content is heavily shaped by security discourses which have consistently stigmatised and targeted Muslims, while downplaying threats from the extreme right 95 .…”

Cognitive assemblages: The entangled nature of algorithmic content moderation

“…Digital platforms must thus relinquish their role as content hosts and develop more active strategies to exclude users that are deemed extremist by public authorities. For this, “companies identify, select, search and interpret suspicious datasets” (Bellanova and de Goede, 2022a: 2). The implementation of such strategies implies the mobilization of a network of cognizers that includes companies, engineers, algorithms, users, terrorism experts, etc., which are articulated through modes of coordination that include community rules and policies, machine learning algorithms, and content flagging features offered to users.…”

“…Rather than collecting evidence, this algorithmic form of governability reinforces preemption in the management of terrorist risk (Bellanova and de Goede, 2022a, 2022b). Platform owners have embraced probabilistic association rules for their security needs.…”

“…Finally, this literature has signaled how the goal of securing in algorithmic security is always emergent, as informed by the available data rather than legal experts (Aradau and Blanke, 2015; Bellanova and de Goede, 2022b). Although algorithms were already a focus of the content moderation literature, existing contributions say little about how algorithms and content moderation became a security practice (Bellanova and de Goede, 2022a). Moderation has mostly been treated as a censorship issue and less as part of a broader set of surveillance and security practices.…”

“…Generally speaking, our approach considers algorithmic content moderation as a heterogeneous set of entities, some of which are social and others technical (Latour, 2005). The literature on content moderation has already highlighted the role of algorithmic and human interventions and the changing nature of power relations between governments, Internet companies, and civil society (Bellanova and de Goede, 2022a, Gillespie, 2018; Roberts, 2019). Hence, to develop our critical approach, it is not enough to point out that algorithms are entangled in human and non-human assemblages.…”