Intrinsic Bias Metrics Do Not Correlate with Application Bias

Abstract

Natural Language Processing (NLP) systems learn harmful societal biases that cause them to amplify inequality as they are deployed in more and more situations. To guide efforts at debiasing these systems, the NLP community relies on a variety of metrics that quantify bias in models. Some of these metrics are intrinsic, measuring bias in word embedding spaces, and some are extrinsic, measuring bias in downstream tasks that the word embeddings enable. Do these intrinsic and extrinsic metrics correlate with each other? We compare intrinsic and extrinsic metrics across hundreds of trained models covering different tasks and experimental conditions. Our results show no reliable correlation between these metrics that holds in all scenarios across tasks and languages. We urge researchers working on debiasing to focus on extrinsic measures of bias, and to make using these measures more feasible via creation of new challenge sets and annotated test data. To aid this effort, we release code, a new intrinsic metric, and an annotated test set focused on gender bias in hate speech. 1

Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond

“…However, there are several applications of the counterfactual data augmentation strategies from §4.1.1 in this setting: for example, Garg et al (2019) construct counterfactuals by swapping lists of "identity terms", with the goal of reducing bias in text classification, and Zhao et al (2018) swap gender markers such as pronouns and names for coreference resolution. Counterfactual data augmentation has also been applied to reduce bias in pre-trained contextualized word embedding models (e.g., Huang et al, 2019;Maudslay et al, 2019) but the extent to which biases in pretrained models propagate to downstream applications remains unclear (Goldfarb-Tarrant et al, 2021). Fairness applications of the distributional criteria discussed in §4.1.2 are relatively rare, but Adragna et al (2020) show that invariant risk minimization (Arjovsky et al, 2019) -which attempts to learn an invariant predictor across multiple "environments" -can reduce the use of spurious correlations with race on the Civil Comments dataset (Borkan et al, 2019) for toxicity detection.…”

Assessing the Reliability of Word Embedding Gender Bias Measures

“…We thus call for future studies on the validity of word embedding bias measures. Fourth, Goldfarb-Tarrant et al (2021) argue that intrinsic (word embeddings) biases sometimes fail to agree with extrinsic biases (measured in downstream tasks, e.g. coreference resolution).…”

Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics

“…A related line of research measures bias present in sentence or word representations (Bolukbasi et al, 2016;Caliskan et al, 2017;Kurita et al, 2019;Sedoc and Ungar, 2019;Chaloner and Maldonado, 2019;Dev and Phillips, 2019;Gonen and Goldberg, 2019;Hall Maudslay et al, 2019;Liang et al, 2020;Shin et al, 2020;Liang et al, 2020;Papakyriakopoulos et al, 2020). However, such intrinsic metrics have been recently shown not to correlate with application bias (Goldfarb-Tarrant et al, 2021). In yet another line of research, Badjatiya et al (2019) detect bias through identifying bias sensitive words.…”

“…Developing such understanding is crucial for drawing reliable conclusions and actionable recommendations regarding bias. We focus on bias measurement for downstream tasks, as Goldfarb-Tarrant et al (2021) have recently shown that there is no reliable correlation between bias measured intrinsically on, for example, word embeddings, and bias measured extrinsically on a downstream task. We narrow down the scope of this paper to tasks that do not involve prediction of a sensitive attribute.…”