Bias Silhouette Analysis: Towards Assessing the Quality of Bias Metrics for Word Embedding Models

Abstract

Word embedding models reflect bias towards genders, ethnicities, and other social groups present in the underlying training data. Metrics such as ECT, RNSB, and WEAT quantify bias in these models based on predefined word lists representing social groups and bias-conveying concepts. How suitable these lists actually are to reveal bias - let alone the bias metrics in general - remains unclear, though. In this paper, we study how to assess the quality of bias metrics for word embedding models. In particular, we present a generic method, Bias Silhouette Analysis (BSA), that quantifies the accuracy and robustness of such a metric and of the word lists used. Given a biased and an unbiased reference embedding model, BSA applies the metric systematically for several subsets of the lists to the models. The variance and rate of convergence of the bias values of each model then entail the robustness of the word lists, whereas the distance between the models' values gives indications of the general accuracy of the metric with the word lists. We demonstrate the behavior of BSA on two standard embedding models for the three mentioned metrics with several word lists from existing research.

A Multibias-mitigated and Sentiment Knowledge Enriched Transformer for Debiasing in Multimodal Conversational Emotion Recognition

“…The recent debiasing models (Bolukbasi et al, 2016;Wang et al, 2020) have only focused on removing gender bias in pre-trained word embeddings, particularly GloVe (Pennington et al, 2014), which has surfaced several social biases (Spliethöver and Wachsmuth, 2021). In this paper, we propose to mitigate five types of biases in GloVe embeddings, i.e., gender, race, religion, age, and LGBTQ+.…”

No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media

“…Using WEAT makes our results comparable with related work. We calculate WEAT scores using the implementation of the WEFE framework (Badilla et al, 2020) and use word lists of Spliethöver and Wachsmuth (2021).…”

“…This impacts generalization performance negatively (Shah et al, 2020) and may have harmful consequences in practical applications (Bender et al, 2021;Joseph and Morgan, 2020). So far, one hurdle to mitigate these problems is the limited reliability of common measures of social bias present in a corpus (Spliethöver and Wachsmuth, 2021), stemming from embedding training algorithms not tailored to low-resource situations (Knoche et al, 2019;Spinde et al, 2021).…”