Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection

Abstract

We present a human-and-model-in-the-loop process for dynamically generating datasets and training better performing and more robust hate detection models. We provide a new dataset of ∼40, 000 entries, generated and labelled by trained annotators over four rounds of dynamic data creation. It includes ∼15, 000 challenging perturbations and each hateful entry has fine-grained labels for the type and target of hate. Hateful entries make up 54% of the dataset, which is substantially higher than comparable datasets. We show that model performance is substantially improved using this approach. Models trained on later rounds of data collection perform better on test sets and are harder for annotators to trick. They also have better performance on HATECHECK, a suite of functional tests for online hate detection. We provide the code, dataset and annotation guidelines for other researchers to use.

Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

“…Our second study focused on collecting ratings for a larger set of posts, but with fewer annota- tors per post to simulate a crowdsourced dataset on toxic language. Drawing from two existing toxic language detection corpora, we select posts that are automatically detected 5 as AAE and/or vulgar from Founta et al ( 2018), and posts that are automatically detected as vulgar and/or annotated as anti-Black from Vidgen et al (2021). 6 Importantly, in this study, we consider anti-Black or AAE posts that could also be vulgar, and allow this vulgarity to cover both potentially offensive identity references (OI) as well as non-identity vulgar words (ONI; see §2.3).…”

“…Anti-Black language denotes racially prejudiced or racist content-subtle (Breitfeller et al, 2019) or overt-which is often a desired target for toxic language detection research (Waseem, 2016;Vidgen et al, 2021). Based on prior work on linking conservative ideologies, endorsement of unrestricted speech, and racial prejudice with reduced likelihood to accept the term "hate speech" (Duckitt and Fisher, 2003;White and Crandall, 2017;Roussos and Dovidio, 2018;Elers and Jayan, 2020), we hypothesize that conservative annotators and those who score highly on the RACISTBELIEFS or FREEOFFSPEECH scales will rate anti-Black tweets as less toxic, and vice-versa.…”

“…the attitudes of annotators towards free speech, racism, and their beliefs on the harms of hate speech, for an accurate estimation of anti-Black language as toxic, offensive, or racist (e.g., by actively taking into consideration annotator ideologies; Waseem, 2016;Vidgen et al, 2021). This can be especially important given that hateful content very often targets marginalized groups and racial minorities (Silva et al, 2016;Sap et al, 2020), and can catalyze violence against them (O' Keeffe et al, 2011;Cleland, 2014).…”

Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks

“…Overall, they are moderately aligned with the prescriptive paradigm. Vidgen et al (2021b) annotate hate speech. They provide annotators with fine-grained definitions for each category as well as very detailed annotation guidelines, and disagreements are resolved by an expert.…”

Retrieve, Discriminate and Rewrite: A Simple and Effective Framework for Obtaining Affective Response in Retrieval-Based Chatbots

“…In order to reduce this risk, one idea is to clean the harmful responses in the dataset, and the other is to detect the harmfulness of the results output by the model. Both of these can be achieved based on some recent works on offensive speech detection (Ranasinghe and Zampieri, 2020) or hate speech detection (Vidgen et al, 2021).…”