Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation

Abstract

Machine learning models are commonly used to detect toxicity in online conversations. These models are trained on datasets annotated by human raters. We explore how raters' self-described identities impact how they annotate toxicity in online comments. We first define the concept of Specialized Rater Pools: rater pools formed based on raters' self-described identities, rather than at random. We formed three such rater pools for this study - specialized rater pools of raters from the U.S. who identify as African American, LGBTQ, and those who identify as neither. Each of these rater pools annotated the same set of comments, which contains many references to these identity groups. We found that rater identity is a statistically significant factor in how raters will annotate toxicity for identity-related annotations. Using preliminary content analysis, we examined the comments with the most disagreement between rater pools and found nuanced differences in the toxicity annotations. Next, we trained models on the annotations from each of the different rater pools, and compared the scores of these models on comments from several test sets. Finally, we discuss how using raters that self-identify with the subjects of comments can create more inclusive machine learning models, and provide more nuanced ratings than those by random raters.

Investigating Labeler Bias in Face Annotation for Machine Learning

“…In this domain, where labelers are annotating portraits of humans, we can conclude that labeler bias exists, depends on labeler demographics, and can be explained using stereotype content [20]. These results are in line with recent findings in CSCW by Goyal et al [24] demonstrating that toxicity labels for online content are influenced by labeler self-identification.…”

“…Prior research has found that even highly experienced labelers fail to produce unbiased labels [29]. Perhaps most relevant to our study is recent work in CSCW by Goyal et al [24] on rater identity. They found that rater identity (i.e., African American, LGBTQ, or neither) significantly influenced how raters annotated toxicity in online comments.…”

“…Investigating labeler bias is essential to understand how labelers influence datasets, but existing studies in this area are limited in scope. Recent work has demonstrated that rater identity plays a significant role in labeling toxicity for online comments [24,50]. One critical paper by Bender et al [5] sheds light on how human biases can be unintentionally perpetuated in machine learning, highlighting that biases introduced in the labeling stage can propagate through to end decisions.…”

Towards Unbiased and Accurate Deferral to Multiple Experts

“…For example, lack of familiarity with the dialect of the content's author can lead to biased decisions [41,10]. Whether the moderator has themself been a target of hate speech [29], or whether their own demographic identity aligns with that being targeted in content they are reviewing [19] can also impact their decisions. Thus, once differences in judging behavior among moderators is acknowledged and accepted, it creates a space for matching different groups of moderators to different content types, based on moderator background (which can be collected via an on-boarding questionnaire).…”

“…We demonstrate the efficacy of our methods via both theoretical analysis ( §2.3) and empirical improvement on synthetic and real-world datasets ( §3 and §4). The latter extends beyond the classic assumption of universal, objective truth to consider recent advocacy for recognizing subjective, community-based gold standards [45,29,17,19].…”