Gender Bias in Masked Language Models for Multiple Languages

Abstract

Masked Language Models (MLMs) pre-trained by predicting masked tokens on large corpora have been used successfully in natural language processing tasks for a variety of languages. Unfortunately, it was reported that MLMs also learn discriminative biases regarding attributes such as gender and race. Because most studies have focused on MLMs in English, the bias of MLMs in other languages has rarely been investigated. Manual annotation of evaluation data for languages other than English has been challenging due to the cost and difficulty in recruiting annotators. Moreover, the existing bias evaluation methods require the stereotypical sentence pairs consisting of the same context with attribute words (e.g. He/She is a nurse). We propose Multilingual Bias Evaluation (MBE) score, to evaluate bias in various languages using only English attribute word lists and parallel corpora between the target language and English without requiring manually annotated data. We evaluated MLMs in eight languages using the MBE and confirmed that gender-related biases are encoded in MLMs for all those languages. We manually created datasets for gender bias in Japanese and Russian to evaluate the validity of the MBE. The results show that the bias scores reported by the MBE significantly correlates with that computed from the above manually created datasets and the existing English datasets for gender bias. * Danushka Bollegala holds concurrent appointments as a Professor at University of Liverpool and as an Amazon Scholar. This paper describes work performed at the University of Liverpool and is not associated with Amazon.

Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples

“…In this paper, we limited our investigation to English PLMs. However, as reported in a lot of previous work, social biases are language independent and omnipresent in PLMs trained for many languages (Kaneko et al, 2022c;Lewis and Lupyan, 2020;Liang et al, 2020;Zhao et al, 2020). We plan to extend this study to cover non-English PLMs in the future.…”

“…The imbalance of gender words in the training data affects the gender bias of a PLM fine-tuned using that data Kaneko et al, 2022c). Using this fact, we propose a method to learn bias-controlled versions of PLMs that express different levels of known gender biases.…”

“…We define rate of bias (r) as the ratio between male and female gendered sentences in a training sample used to fine-tune a PLM. A PLM fine-tuned mostly on male sentences is likely to generate sentences containing mostly male words, while a PLM finetuned on female sentences is likely to generate sentences containing mostly female words Kaneko et al, 2022c). Therefore, an accurate intrinsic bias evaluation measure is expected to return a score indicating a bias towards the male gender for a male bias-controlled PLM, while it is expected to return a score indicating a bias towards the female gender for a female bias-controlled PLM.…”

Fairness in Language Models Beyond English: Gaps and Challenges

“…Milios and BehnamGhader (2022); España-Bonet and Barrón-Cedeño (2022) illustrate the inefficiency of direct translation methods, and España-Bonet and Barrón-Cedeño (2022) advocates for the creation of culturally-sensitive datasets for fairness assessment. However, Kaneko et al (2022) proposes a way to generate parallel corpora for other languages that bears high correlation with human bias annotations.…”

“…This is particularly relevant, as in a practical setting, treating identities as composites of various demographic attributes is a necessity. Kaneko et al (2022) measures gender bias in masked language models and proposes a method to use parallel corpora to evaluate bias in languages shown to have high correlations with human bias annotations. In cases where manually annotated data doesn't exist, this could prove helpful.…”