Crossmod: A Cross-Community Learning-based System to Assist Reddit Moderators

Abstract

In this paper, we introduce a novel sociotechnical moderation system for Reddit called Crossmod. Through formative interviews with 11 active moderators from 10 different subreddits, we learned about the limitations of currently available automated tools, and how a new system could extend their capabilities. Developed out of these interviews, Crossmod makes its decisions based on cross-community learning---an approach that leverages a large corpus of previous moderator decisions via an ensemble of classifiers. Finally, we deployed Crossmod in a controlled environment, simulating real-time conversations from two large subreddits with over 10M subscribers each. To evaluate Crossmod's moderation recommendations, 4 moderators reviewed comments scored by Crossmod that had been drawn randomly from existing threads. Crossmod achieved an overall accuracy of 86% when detecting comments that would be removed by moderators, with high recall (over 87.5%). Additionally, moderators reported that they would have removed 95.3% of the comments flagged by Crossmod; however, 98.3% of these comments were still online at the time of this writing (i.e., not removed by the current moderation system). To the best of our knowledge, Crossmod is the first open source, AI-backed sociotechnical moderation system to be designed using participatory methods.

PolicyKit: Building Governance in Online Communities

“…In many cases, these automation tools are not, or do not start out, embedded in a platform but grow separately as one-off bespoke pieces of software [22], making them more difficult to author and manage. Additional tools exist that help mods create better policies [8,52] but still are focused on the role of mods. Finally, some tools exist that are aimed at regular members of a community [23,32,46], but these are primarily only used to enact policies for an individual as opposed to a group.…”

Reconsidering Self-Moderation

“…As of early 2016, none of the 15 major platforms they analyzed provided a specific definition for harassment in any of the 56 documents they collected, and only Twitter and Instagram provided descriptions of behaviors that were considered when determining whether actions would be defined as harassment. 11 In complementary work, West studied users' reactions to content removal and folk theories about how those systems work, noting that users are often left to speculate about reasons for removal due to a general lack of transparency in moderation actions [103]. Suzor et al proposed specific ways in which increased transparency could help educate users and establish a sense of trust in these processes [97].…”

“…Chandrasekharan et al avoid defining what behaviors are "problematic" by using external communities' definitions of problematic behavior to define rules for a new community. This results in a classifier that works not from consciously defined rules but from the aggregation of prior users' moderation decisions [11]. Chandrasekharan et al take an important step by choosing to use this classifier in a way that brings comments to the attention of moderators rather than removing the comments without human oversight.…”

“…The former approach can be influenced by significant bias from both the labelers and the source of the training data [6,43], with offenses specific to the context of the host community unlikely to be captured by a general dataset. In this sense, Chandrasekharan et al [11,13] have achieved some success by allowing moderators to build from the experiences of other communities, but questions of bias remain. The latter approach, drawing data from past decisions in the host community, requires a significant prior body of decisions and is unlikely to proactively predict changes in how content would be moderated due to external circumstances (e.g., a new type of problematic behavior appearing or a social change in the implicit meaning of a particular phrase or term); as machine learning models are trained on prior data, they are mathematically disadvantaged in making predictions on cases not contained in that data.…”

“…Chandrasekharan et al's Crossmod system [11] is a good example of work that thoughtfully considers ethical and technical tradeoffs; the system uses data from multiple communities on Reddit to train models for moderation, and allows moderators to customize various high-level aspects of the model to find a better fit for their community. Though the model itself remains somewhat opaque, the options for customization allow more supervision than a traditional externally-trained and blanket-applied model.…”