How Gender Debiasing Affects Internal Model Representations, and Why It Matters

Abstract

Common studies of gender bias in NLP focus either on extrinsic bias measured by model performance on a downstream task or on intrinsic bias found in models' internal representations. However, the relationship between extrinsic and intrinsic bias is relatively unknown. In this work, we illuminate this relationship by measuring both quantities together: we debias a model during downstream fine-tuning, which reduces extrinsic bias, and measure the effect on intrinsic bias, which is operationalized as bias extractability with information-theoretic probing. Through experiments on two tasks and multiple bias metrics, we show that our intrinsic bias metric is a better indicator of debiasing than (a contextual adaptation of) the standard WEAT metric, and can also expose cases of superficial debiasing. Our framework provides a comprehensive perspective on bias in NLP models, which can be applied to deploy NLP systems in a more informed manner. 1 * Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion.

Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)

“…Our case study extends the experiments done by Orgad et al (2022). In their work, they tested a RoBERTa-based classifier finetuned on Bias in Bios.…”

“…Winobias sentences consist of an anti-and a pro-stereotypical sentence, as shown in Figure 1. Coreference systems should be able to resolve both sentences correctly, but most perform poorly on the anti-stereotypical ones Orgad et al, 2022).…”

“…Winobias was originally proposed as an extrinsic evaluation dataset, with a reported metric of anti-and pro-stereotypical performance disparity. However, other metrics can also be measured, both intrinsic and extrinsic, as shown in several studies Nangia et al, 2020b;Orgad et al, 2022). For example, one can measure how many stereotypical choices the model preferred over anti-stereotypical choices (an extrinsic performance measure), as done on Winogender , a similar dataset.…”

“…Here we train the same model on the same training set, but evaluate it on three types of test sets: the original test set alongside balanced versions of it, which have equal numbers of females and males in every profession, by either subsampling or oversampling. 2 We follow Orgad et al (2022) and report nine different metrics on this task, measuring either some notion of performance gap or a statistical metric from the fairness literature. For details on the metrics measured in this experiments, see Appendix C.…”

“…It is critical to consider the impact of the data used when measuring intrinsic bias metrics on a language model. Previous work Cao et al, 2022;Orgad et al, 2022) inspected the correlations between extrinsic and in- Figure 3: Correlation between an intrinsic metric (compression) and an extrinsic metric (sufficiency gap sums), for various models trained on occupation prediction task. "None" was trained on the original dataset, "Oversampling" was trained on an oversampled dataset, "Subsampling" was trained on a subsampled dataset and "Scrubbing" was trained on a scrubbed dataset (explicit gender words like "he" and "she" were removed).…”