Fight Fire with Fire: Fine-tuning Hate Detectors using Large Samples of Generated Hate Speech

Abstract

Automatic hate speech detection is hampered by the scarcity of labeled datasetd, leading to poor generalization. We employ pretrained language models (LMs) to alleviate this data bottleneck. We utilize the GPT LM for generating large amounts of synthetic hate speech sequences from available labeled examples, and leverage the generated data in fine-tuning large pretrained LMs on hate detection. An empirical study using the models of BERT, RoBERTa and ALBERT, shows that this approach improves generalization significantly and consistently within and across data distributions. In fact, we find that generating relevant labeled hate speech sequences is preferable to using out-of-domain, and sometimes also within-domain, human-labeled examples.

Free speech or Free Hate Speech? Analyzing the Proliferation of Hate Speech in Parler

“…Recently, Transformer-based architectures (Mozafari et al, 2019;Aluru et al, 2020;Samghabadi et al, 2020;Salminen et al, 2020;Qian et al, 2021;Kennedy et al, 2020;Arviv et al, 2021) achieved significant improvements over RNN and CNN models (Zhang et al, 2016;Gambäck and Sikdar, 2017;Del Vigna12 et al, 2017;Park and Fung, 2017). In an effort to mitigate the need for extensive annotation some works use transformers to generate more samples, e.g., (Vidgen et al, 2020b;Wullach et al, 2020Wullach et al, , 2021. Zhou et al (2021) integrate features from external resources to support the model performance.…”