Backtranslation Feedback Improves User Confidence in MT, Not Quality

Abstract

Translating text into a language unknown to the text's author, dubbed outbound translation, is a modern need for which the user experience has significant room for improvement, beyond the basic machine translation facility. We demonstrate this by showing three ways in which user confidence in the outbound translation, as well as its overall final quality, can be affected: backward translation, quality estimation (with alignment) and source paraphrasing. In this paper, we describe an experiment on outbound translation from English to Czech and Estonian. We examine the effects of each proposed feedback module and further focus on how the quality of machine translation systems influence these findings and the user perception of success. We show that backward translation feedback has a mixed effect on the whole process: it increases user confidence in the produced translation, but not the objective quality.

Beyond General Purpose Machine Translation: The Need for Context-specific Empirical Research to Design for Appropriate User Trust

“…To circumvent this challenge, users and researchers have relied heavily on back-translation (translating an output Beyond General Purpose Machine Translation: The Need for Context-specific Empirical Research to Design for Appropriate User Trust 3 back to the source language) [32,33,46,63]. While this offers some insight into the translation in a language the user can understand, it remains unclear how reliable this strategy is in practice, or in what specific cases it helps or fails to support users in calibrating their trust in MT [33,67]. For example, Zouhar et al found that showing users the backtranslation increased their trust in an MT model, but did not help them to identify or improve poor quality translations [67].…”

“…While this offers some insight into the translation in a language the user can understand, it remains unclear how reliable this strategy is in practice, or in what specific cases it helps or fails to support users in calibrating their trust in MT [33,67]. For example, Zouhar et al found that showing users the backtranslation increased their trust in an MT model, but did not help them to identify or improve poor quality translations [67]. This is consistent with evidence in MT and in other ML domains that showing additional interpretability or explainability information can increase users' trust in a model simply because the additional information is there, even if that information indicates that the model may be incorrect [20,33].…”

Opportunities for Human-centered Evaluation of Machine Translation Systems

“…This will require new designs and evaluation methods. For example, Zouhar et al (2021) found that providing back-translations can improve user trust in outbound translations but did not correspond to an improvement in translation quality. As detailed in Section 3.3.2, careful task design will be necessary to prove the value of these affordances.…”

AuGPT: Auxiliary Tasks and Data Augmentation for End-To-End Dialogue with Pre-Trained Language Models

“…We use two different Transformer-based machine translation systems to paraphrase our data. We used Edunov et al (2018)'s system with French and the system of Macháček et al (2020); Zouhar et al (2021) with additional 40 pivot languages. Based on empirical analysis of translation quality, we chose 10 pivot languages for our data -we obtain 10 different paraphrases for each input utterance.…”