Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech

Abstract

Undermining the impact of hateful content with informed and non-aggressive responses, called counter narratives, has emerged as a possible solution for having healthier online communities. Thus, some NLP studies have started addressing the task of counter narrative generation. Although such studies have made an effort to build hate speech / counter narrative (HS/CN) datasets for neural generation, they fall short in reaching either highquality and/or high-quantity. In this paper, we propose a novel human-in-the-loop data collection methodology in which a generative language model is refined iteratively by using its own data from the previous loops to generate new training samples that experts review and/or post-edit. Our experiments comprised several loops including dynamic variations. Results show that the methodology is scalable and facilitates diverse, novel, and cost-effective data collection. To our knowledge, the resulting dataset is the only expertbased multi-target HS/CN dataset available to the community.

Hate Speech Criteria: A Modular Approach to Task-Specific Hate Speech Definitions

“…Mathew et al (2021) are the only ones to mention Caucasian as a target group, which we mark with a '-' to signal that this paper is explicit about not restricting itself to non-dominant groups. Most papers with a ✓for ND specifically define their targets to apply to nondominant groups only (Chung et al, 2019;Basile et al, 2019;Fanton et al, 2021), with the exception of Talat and Hovy (2016), who mention minorities.…”

Perspectives on Incorporating Expert Feedback into Model Updates

“…Adding data [16,45,51,121] Relabeling data [76] Reweighting data [12,64,137] Collecting expert labels [98] Passive observation [69,84,118]…”

“…Data collection may transform D into D by asking the expert to approve the weight placed on [12] or to provide a label for a datapoint [76]. Experts may also review new datapoints, where each new datapoint x is selected by some heuristic (e.g., high uncertainty regions) and the corresponding y is specified by the expert [45,121]. Some work considers collecting multiple labels from various experts for each x [98].…”

SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets

“…However the approach was deemed inferior to a fully manual approach due to data quality issues with the synthetic dataset. Finally, Fanton et al [17] propose a data collection method where a language model is iteratively finetuned on human-edited versions of its own generations.…”