“You Sound Just Like Your Father” Commercial Machine Translation Systems Include Stylistic Biases

Hovy, Dirk; Bianchi, Federico; Fornaciari, Tommaso

doi:10.18653/v1/2020.acl-main.154

Cited by 40 publications

(34 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…To address gender bias in neural network approaches to coreference resolution Rudinger et al (2018); Zhao et al (2018) 2018)). Hovy et al (2020) show that this overamplification consistently makes translations sound older and more male than the original authors.…”

Section: Countermeasuresmentioning

confidence: 75%

Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview

Shah

Schwartz

Hovy

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

Self Cite

132

View full text Add to dashboard Cite

An increasing number of natural language processing papers address the effect of bias on predictions, introducing mitigation techniques at different parts of the standard NLP pipeline (data and models). However, these works have been conducted individually, without a unifying framework to organize efforts within the field. This situation leads to repetitive approaches, and focuses overly on bias symptoms/effects, rather than on their origins, which could limit the development of effective countermeasures. In this paper, we propose a unifying predictive bias framework for NLP. We summarize the NLP literature and suggest general mathematical definitions of predictive bias. We differentiate two consequences of bias: outcome disparities and error disparities, as well as four potential origins of biases: label bias, selection bias, model overamplification, and semantic bias. Our framework serves as an overview of predictive bias in NLP, integrating existing work into a single structure, and providing a conceptual baseline for improved frameworks.

show abstract

Section: Countermeasuresmentioning

confidence: 75%

Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview

Shah

Schwartz

Hovy

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

Self Cite

132

View full text Add to dashboard Cite

show abstract

“…Gender affects myriad aspects of NLP, including corpora, tasks, algorithms, and systems Costa-jussà, 2019;Sun et al, 2019). For example, statistical gender biases are rampant in word embeddings (Jurgens et al, 2012;Bolukbasi et al, 2016;Caliskan et al, 2017;Garg et al, 2018;Zhao et al, 2018b;Basta et al, 2019;Chaloner and Maldonado, 2019;Du et al, 2019;Ethayarajh et al, 2019;Kaneko and Bollegala, 2019;Kurita et al, 2019;-including multilingual ones (Escudé Font and Costa-jussà, 2019;Zhou et al, 2019)-and affect a wide range of downstream tasks including coreference resolution (Zhao et al, 2018a;Cao and Daumé III, 2020;Emami et al, 2019), part-ofspeech and dependency parsing (Garimella et al, 2019), language modeling (Qian et al, 2019;Nangia et al, 2020), appropriate turn-taking classification (Lepp, 2019), relation extraction (Gaut et al, 2020), identification of offensive content (Sharifirad and Matwin, 2019;, and machine translation (Stanovsky et al, 2019;Hovy et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

Multi-Dimensional Gender Bias Classification

Dinan¹,

Fan²,

Wu³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Machine learning models are trained to find patterns in data. NLP models can inadvertently learn socially undesirable patterns when training on gender biased text. In this work, we propose a novel, general framework that decomposes gender bias in text along several pragmatic and semantic dimensions: bias from the gender of the person being spoken about, bias from the gender of the person being spoken to, and bias from the gender of the speaker. Using this fine-grained framework, we automatically annotate eight large scale datasets with gender information. In addition, we collect a new, crowdsourced evaluation benchmark. Distinguishing between gender bias along multiple dimensions enables us to train better and more fine-grained gender bias classifiers. We show our classifiers are valuable for a variety of applications, like controlling for gender bias in generative models, detecting gender bias in arbitrary text, and classifying text as offensive based on its genderedness.

show abstract

“…Style itself is a broad concept (Kang and Hovy, 2019). It includes both simple high-level stylistic aspects of language such as verbosity (Marchisio et al, 2019;Agrawal and Carpuat, 2019;Lakew et al, 2019), formality (Niu et al, 2017;Xu et al, 2019), politeness and complex aspects such as demography (Vanmassenhove et al, 2018;Moryossef et al, 2019;Hovy et al, 2020) and personal traits (Mirkin and Meunier, 2015;Rabinovich et al, 2017;Michel and Neubig, 2018).…”

Section: Related Workmentioning

confidence: 99%

“…Following the common practice in evaluating the style imitation (e.g. see (Michel and Neubig, 2018;Hovy et al, 2020)), we train a classifier to predict the translator style of the output of various models. We employ a Logistic Regression classifier based on both uni-gram and bi-gram word features.…”

Section: Style Imitationmentioning

confidence: 99%

Towards Modeling the Style of Translators in Neural Machine Translation

Wang¹,

Cuong²,

Federico³

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

One key ingredient of neural machine translation is the use of large datasets from different domains and resources (e.g. Europarl, TED talks). These datasets contain documents translated by professional translators using different but consistent translation styles. Despite that, the model is usually trained in a way that neither explicitly captures the variety of translation styles present in the data nor translates new data in different and controllable styles. In this work, we investigate methods to augment the state-of-the-art Transformer model with translator information that is available in part of the training data. We show that our style-augmented translation models are able to capture the style variations of translators and to generate translations with different styles on new data. Indeed, the generated variations differ significantly, up to +4.5 BLEU score difference. Despite that, human evaluation confirms that the translations are of the same quality. * Y. Wang carried out this work during an internship with Amazon AI.

show abstract

“You Sound Just Like Your Father” Commercial Machine Translation Systems Include Stylistic Biases

Cited by 40 publications

References 9 publications

Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview

Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview

Multi-Dimensional Gender Bias Classification

Towards Modeling the Style of Translators in Neural Machine Translation

Contact Info

Product

Resources

About