XHate-999: Analyzing and Detecting Abusive Language Across Domains and Languages

Abstract

We present XHATE-999, a multi-domain and multilingual evaluation data set for abusive language detection. By aligning test instances across six typologically diverse languages, XHATE-999 for the first time allows for disentanglement of the domain transfer and language transfer effects in abusive language detection. We conduct a series of domain-and language-transfer experiments with state-of-the-art monolingual and multilingual transformer models, setting strong baseline results and profiling XHATE-999 as a comprehensive evaluation resource for abusive language detection. Finally, we show that domain-and language-adaptation, via intermediate masked language modeling on abusive corpora in the target language, can lead to substantially improved abusive language detection in the target language in the zero-shot transfer setups.

Towards generalisable hate speech detection: a review on obstacles and solutions

“…In Swamy, Jamatia & Gambäck (2019) ’s study with fine-tuned BERT models ( Devlin et al, 2019 ), Founta and OLID produced models that performed well on each other. The source of such differences are usually traced back to search terms ( Swamy, Jamatia & Gambäck, 2019 ), topics covered ( Nejadgholi & Kiritchenko, 2020 ; Pamungkas, Basile & Patti, 2020 ), label definitions ( Pamungkas & Patti, 2019 ; Pamungkas, Basile & Patti, 2020 ; Fortuna, Soler-Company & Wanner, 2021 ), and data source platforms ( Glavaš, Karan & Vulić, 2020 ; Karan & Šnajder, 2018 ).…”

“…The ideal case would be to be able to use data in one language for training and apply the model on data in another language, which would help address the challenge in low-resource languages. In a few studies ( Pamungkas, Basile & Patti, 2020 ; Glavaš, Karan & Vulić, 2020 ; Arango, Prez & Poblete, 2020 ; Fortuna, Soler-Company & Wanner, 2021 ), language was included as a separate variable, alongside a “domain” variable independent to it, which is characterised by the source platform or the data collection method. These cross-lingual experiments are summarised in Table 3 .…”

“…In terms of models , pre-trained multilingual word embeddings (MUSE ( Conneau et al, 2017 )) and language models (mBERT ( Devlin et al, 2019 ), XLM-R ( Conneau et al, 2020 )) are frequently chosen as baselines. They are an intuitive and easily accessible starting point for cross-lingual experiments, but their limitations are also clear—the “curse of multilinguality” trades off single-language performance for its broad language coverage, as displayed in the results of the cross-lingual generalisation studies mentioned above ( Pamungkas & Patti, 2019 ; Pamungkas, Basile & Patti, 2020 ; Glavaš, Karan & Vulić, 2020 ; Arango, Prez & Poblete, 2020 ; Fortuna, Soler-Company & Wanner, 2021 ) and in other tasks ( Conneau et al, 2020 ). Similarly to the monolingual case, there are cases where traditional machine learning models outperform deep learning ones, such as SVM ( Pamungkas, Basile & Patti, 2020 ) and GBDT ( Arango, Prez & Poblete, 2020 ) compared to LSTM.…”

“…When it comes to the data , the most prominent additional factor compared to the monolingual setting is the similarity between the training (source) and testing (target) languages. For instance, Among the wide range of languages that Glavaš, Karan & Vulić (2020) have tested, the cross-lingual performance drop between English, the source language, and German, the most similar target language, was less than one third of that between English and Turkish, when using mBERT on Wulczyn .…”

“…Although these studies more or less consider the “language” and “domain” variables as separate, there exists evidence that the two types of generalisation interact with each other. Studies that control the language variable more carefully tend to show a smaller drop across languages—for example, by manually translating exactly the same data ( Glavaš, Karan & Vulić, 2020 ), as opposed to using automatic translation ( Pamungkas & Patti, 2019 ; Pamungkas, Basile & Patti, 2020 ; Arango, Prez & Poblete, 2020 ) or different language dataset from the same shared task ( Pamungkas & Patti, 2019 ; Pamungkas, Basile & Patti, 2020 ; Fortuna, Soler-Company & Wanner, 2021 ). Furthermore, adding data from a different domain can act as a regulariser from overfitting to the training language ( Glavaš, Karan & Vulić, 2020 ).…”