Operationalising ‘toxicity’ in the manosphere: Automation, platform governance and community health

Abstract

Social media platforms have been struggling to moderate at scale. In an effort to better cope with content moderation discussion has turned to the role that automated machine-learning (ML) tools might play. The development of automated systems by social media platforms is a notoriously opaque process and public values that pertain to the common good are at stake within these often-obscured processes. One site in which social values are being negotiated is in the framing of what is considered ‘toxic’ by platforms in the development of automated moderation processes. This study takes into consideration differing notions of toxicity – community, platform and societal by examining three measures of toxicity and community health (the ML tool Perspective API; Reddit’s 2020 Content Policy; and the Sense of Community Index-2) and how they are operationalised in the context of r/MGTOW – an antifeminist group known for its misogyny. Several stages of content analysis were conducted on the top posts and comments in r/MGTOW to examine how these different measures of toxicity operate. This paper provides insight into the logics and technicalities of automated moderation tools, platform governance structures, and frameworks for understanding community metrics to interrogate existing uses of ‘toxicity’ as applied to cultural or social subcommunities online. We make a distinction between two used terms: civility and toxicity. Our analysis points to a tension between current social framings and operationalised notions of ‘toxicity’. We argue that there is a clear distinction between civility and toxicity – incivility is a measure of internal perceptions of harm within a community, whereas toxicity is a measure of the capacity for social harms outside of the bounds of the community. This nuanced understanding will enable more targeted interventions to be developed to destabilise the internal conditions that make groups like r/MGTOW internally ‘healthy’ yet externally toxic.