A Comparative Study on Word Embeddings and Social NLP Tasks

Abstract

In recent years, grey social media platforms, those with a loose moderation policy on cyberbullying, have been attracting more users. Recently, data collected from these types of platforms have been used to pre-train word embeddings (social-media-based), yet these word embeddings have not been investigated for social NLP related tasks. In this paper, we carried out a comparative study between social-mediabased and non-social-media-based word embeddings on two social NLP tasks: Detecting cyberbullying and Measuring social bias. Our results show that using social-media-based word embeddings as input features, rather than non-social-media-based embeddings, leads to better cyberbullying detection performance. We also show that some word embeddings are more useful than others for categorizing offensive words. However, we do not find strong evidence that certain word embeddings will necessarily work best when identifying certain categories of cyberbullying within our datasets. Finally, We show even though most of the state-of-the-art bias metrics ranked social-media-based word embeddings as the most socially biased, these results remain inconclusive and further research is required.

Darkness can not drive out darkness: Investigating Bias in Hate SpeechDetection Models

“…Then, I investigate the performance of Bi-LSTM model with an un-trainable embeddings layer of the five word embeddings on the used five hate-speechrelated datasets. The results indicate that the word embeddings that are pre-trained on biased datasets social-media-based, outperform the other word embeddings that are trained on informational data, informational-based on the tasks of offenses categorization and hate speech detection (Elsafoury et al, 2022b).…”

“…In (Elsafoury et al, 2022a;Elsafoury, 2023), I investigate how the hateful content on social media and other platforms that are used to collect data and pre-train NLP models, is being encoded by those NLP models to form systematic offensive stereotyping (SOS) bias against marginalized groups of people. Especially with imbalanced representation and co-occurrence of the hateful content with the marginalized identity groups.…”

“…I introduce the systematic offensive stereotyping (SOS) bias. I formally define it and propose a method to measure it and validate it in static (Elsafoury et al, 2022a) and contextual word embeddings (Elsafoury et al, 2022a). Finally, I study how it impacts the performance of these word embeddings on hate speech detection models.…”

“…Instead of using crowdsourced sentence pairs that express socially bias sentences and socially unbiased sentences, I use synthesized sentence pairs that express profane sentences and non-profane sentence-pairs. I measure the SOS bias scores in 15 static word embeddings (Elsafoury et al, 2022a) and 3 contextual word embeddings (Elsafoury, 2023). The results show that for static word embeddings, there is SOS bias in all the inspected word embeddings, and it is significantly higher towards marginalized groups.…”

Thesis Distillation: Investigating The Impact of Bias in NLP Models on Hate Speech Detection

“…(Elsafoury, 2022). I propose a method to measure it and validate it in static (Elsafoury et al, 2022a) and contextual word embeddings (Elsafoury et al, 2022a). Finally, I study how it impacts the performance of these word embeddings on hate speech detection models.…”