HABERTOR: An Efficient and Effective Deep Hatespeech Detector

Abstract

We present our HABERTOR model for detecting hatespeech in large scale user-generated content. Inspired by the recent success of the BERT model, we propose several modifications to BERT to enhance the performance on the downstream hatespeech classification task. HABERTOR inherits BERT's architecture, but is different in four aspects: (i) it generates its own vocabularies and is pre-trained from the scratch using the largest scale hatespeech dataset; (ii) it consists of Quaternionbased factorized components, resulting in a much smaller number of parameters, faster training and inferencing, as well as less memory usage; (iii) it uses our proposed multisource ensemble heads with a pooling layer for separate input sources, to further enhance its effectiveness; and (iv) it uses a regularized adversarial training with our proposed finegrained and adaptive noise magnitude to enhance its robustness. Through experiments on the large-scale real-world hatespeech dataset with 1.4M annotated comments, we show that HABERTOR works better than 15 state-ofthe-art hatespeech detection methods, including fine-tuning Language Models. In particular, comparing with BERT, our HABERTOR is 4∼5 times faster in the training/inferencing phase, uses less than 1/3 of the memory, and has better performance, even though we pretrain it by using less than 1% of the number of words. Our generalizability analysis shows that HABERTOR transfers well to other unseen hatespeech datasets and is a more efficient and effective alternative to BERT for the hatespeech classification.

Tackling racial bias in automated online hate detection: Towards fair and accurate detection of hateful users with geometric deep learning

“…Hateful content detection has been extensively studied, with numerous papers presenting new datasets, models and architectures for hateful content detection, including several subtasks, such as multilingual, multimodal and multi-target hate detection [29,61,68]. Notably, incorporating sentence-level context has been shown to improve hateful content detection, with several papers deploying transformer-based models such as Bidirectional Encoder Representation from Transformers (BERT) to better distinguish between hateful and non-hateful content, even when they have lexical similarities [37,57]. Indeed, as Mullah and Zainon [46] write in their comprehensive review of ML methods for automated hate speech detection, deep learning techniques leveraging such language models can considerably improve how context-dependent hate speech is detected.…”

How Robust are Fact Checking Systems on Colloquial Claims?

“…Other works have attempted to mitigate biases, such as gender bias (Dinan et al, 2020a) and racial bias (Sap et al, 2019). Recently, Tran et al (2020) modify BERT (Devlin et al, 2019) to detect hatespeech. introduce a method to distill safety standards into the generative dialogue agent.…”

Character-level HyperNetworks for Hate Speech Detection

“…Another challenge of practical importance is that state-of-the-art deep learning networks are extremely large, reaching up to hundreds of millions, or billions of parameters. Ideally, hate detection would be performed in real time at the end device, allowing the alert the user prior to posting hateful content, for example, but high-performing compact hate detection models are required to achieve this goal (Tran et al, 2020;Behzadi et al, 2021).…”