Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort

Mendonça, Vânia; Rei, Ricardo; Coheur, Luísa; Sardinha, Alberto; Santos, Ana Lúcia

doi:10.18653/v1/2021.acl-long.242

Cited by 3 publications

(11 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The bandit algorithms learned from feedback simulated using a sentence-level BLEU score (Papineni et al 2002) between the selected automatic translation and a reference translation. Another one is a previous work of ours (Mendonça et al 2021), in which we framed the problem of dynamically converging to the performance of the best individual MT system as a problem of prediction with expert advice (when feedback is available to the translations outputted by all the systems in the ensemble) and adversarial multi-armed bandits (Robbins 1952;Lai and Robbins 1985) (when feedback is only available for the final translation chosen by the ensemble). We simulated the human-inthe-loop by using actual human ratings obtained from an MT shared task (Barrault et al 2019), when available in the data, and proposed different fallback strategies to cope with the lack of human ratings for some of the translations.…”

Section: Online Learning For Machine Translationmentioning

confidence: 99%

“…We thus build upon our previous work (Mendonça et al 2021), since it is the only one that takes advantage of human quality ratings rather than translations or post-edits. This setting not only reduces the human effort involved in improving the MT ensemble, but it also should prove to be more suitable to represent real world MT scenarios, such as Web translation systems or MT shared tasks, in which the human-in-the-loop is not expected to provide a translation.…”

Section: Online Learning For Machine Translationmentioning

confidence: 99%

“…In this article, we build on our previous work in which we combined online learning and MT (Mendonça et al 2021), as described in Section 3.2. An overview of the online learning scenario is illustrated in Fig.…”

Section: Problem Definitionmentioning

confidence: 99%

“…Overview of the online learning process applied to MT, at each iteration t. The grey dashed arrows represent flows that only occur when using prediction with expert advice (Mendonça et al 2021).…”

Section: Figurementioning

confidence: 99%

“…To fill this gap, we build on our previous work (Mendonça et al 2021), in which we leveraged on human ratings to learn the weights of an ensemble of arbitrary MT models in an online fashion, in order to dynamically improve its performance for the language pairs in hand. We extend this proposal by incorporating model-independent active learning strategies that decide, for each pair source-translations, whether it is worthy to query the user for a rating, thus reducing the effort needed to improve the performance of the MT ensemble.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Onception: Active Learning with Expert Advice for Real World Machine Translation

Mendonça¹,

Rei²,

Coheur³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Active learning can play an important role in low-resource settings (i.e., where annotated data is scarce), by selecting which instances may be more worthy to annotate. Most active learning approaches for Machine Translation assume the existence of a pool of sentences in a source language, and rely on human annotators to provide translations or post-edits, which can still be costly. In this article, we assume a real world human-in-the-loop scenario in which: (1) the source sentences may not be readily available, but instead arrive in a stream; (2) the automatic translations receive feedback in the form of a rating, instead of a correct/edited translation, since the human-in-the-loop might be a user looking for a translation, but not be able to provide one. To tackle the challenge of deciding whether each incoming pair source-translations is worthy to query for human feedback, we resort to a number of stream-based active learning query strategies. Moreover, since we not know in advance which query strategy will be the most adequate for a certain language pair and set of Machine Translation models, we propose to dynamically combine multiple strategies using prediction with expert advice. Our experiments show that using active learning allows to converge to the best Machine Translation systems with fewer human interactions. Furthermore, combining multiple strategies using prediction with expert advice often outperforms several individual active learning strategies with even fewer interactions.

show abstract

Section: Online Learning For Machine Translationmentioning

confidence: 99%

Section: Online Learning For Machine Translationmentioning

confidence: 99%

Section: Problem Definitionmentioning

confidence: 99%

Section: Figurementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Onception: Active Learning with Expert Advice for Real World Machine Translation

Mendonça¹,

Rei²,

Coheur³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Onception: Active Learning with Expert Advice for Real World Machine Translation

Mendonça

Rei

Coheur

et al. 2023

Computational Linguistics

View full text Add to dashboard Cite

Active learning can play an important role in low-resource settings (i.e., where annotated data is scarce), by selecting which instances may be more worthy to annotate. Most active learning approaches for Machine Translation assume the existence of a pool of sentences in a source language, and rely on human annotators to provide translations or post-edits, which can still be costly. In this article, we apply active learning to a real world human-in-the-loop scenario in which we assume that: (1) the source sentences may not be readily available, but instead arrive in a stream; (2) the automatic translations receive feedback in the form of a rating, instead of a correct/edited translation, since the human-in-the-loop might be a user looking for a translation, but not be able to provide one. To tackle the challenge of deciding whether each incoming pair source–translations is worthy to query for human feedback, we resort to a number of stream-based active learning query strategies. Moreover, since we do not know in advance which query strategy will be the most adequate for a certain language pair and set of Machine Translation models, we propose to dynamically combine multiple strategies using prediction with expert advice. Our experiments on different language pairs and feedback settings show that using active learning allows to converge to the best Machine Translation systems with fewer human interactions. Furthermore, combining multiple strategies using prediction with expert advice outperforms several individual active learning strategies with even fewer interactions, particularly in partial feedback settings.

show abstract

Toward More Effective Human Evaluation for Machine Translation

Saldías¹,

Foster²,

Freitag³

et al. 2022

Preprint

View full text Add to dashboard Cite

Improvements in text generation technologies such as machine translation have necessitated more costly and time-consuming human evaluation procedures to ensure an accurate signal. We investigate a simple way to reduce cost by reducing the number of text segments that must be annotated in order to accurately predict a score for a complete test set. Using a sampling approach, we demonstrate that information from document membership and automatic metrics can help improve estimates compared to a pure random sampling baseline. We achieve gains of up to 20% in average absolute error by leveraging stratified sampling and control variates. Our techniques can improve estimates made from a fixed annotation budget, are easy to implement, and can be applied to any problem with structure similar to the one we study.

show abstract

Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort

Cited by 3 publications

References 31 publications

Onception: Active Learning with Expert Advice for Real World Machine Translation

Onception: Active Learning with Expert Advice for Real World Machine Translation

Onception: Active Learning with Expert Advice for Real World Machine Translation

Toward More Effective Human Evaluation for Machine Translation

Contact Info

Product

Resources

About