APE at Scale and Its Implications on MT Evaluation Biases

Freitag, Markus; Caswell, Isaac; Roy, S.

doi:10.18653/v1/w19-5204

Cited by 48 publications

(61 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Our model can be regarded as an automatic postediting system -a system designed to fix systematic MT errors that is decoupled from the main MT system. Automatic post-editing has a long history, including rule-based (Knight and Chander, 1994), statistical (Simard et al, 2007) and neural approaches (Junczys-Dowmunt and Grundkiewicz, 2016;Pal et al, 2016;Freitag et al, 2019). In terms of architectures, modern approaches use neural sequence-to-sequence models, either multi-source architectures that consider both the original source and the baseline translation (Junczys-Dowmunt and Grundkiewicz, 2016;Pal et al, 2016), or monolingual repair systems, as in Freitag et al (2019), which is concurrent work to ours.…”

Section: Automatic Post-editingmentioning

confidence: 98%

“…For training, the DocRepair model only requires monolingual document-level data. While we create synthetic training data via round-trip translation similarly to earlier work (Junczys-Dowmunt and Grundkiewicz, 2016;Freitag et al, 2019), note that we purposefully use sentence-level MT systems for this to create the types of consistency errors that we aim to fix with the context-aware DocRepair model. Not all types of consistency errors that we want to fix emerge from a round-trip translation, so access to parallel document-level data can be useful (Section 6.2).…”

Section: Automatic Post-editingmentioning

confidence: 99%

See 1 more Smart Citation

Context-Aware Monolingual Repair for Neural Machine Translation

Voita¹,

Sennrich²,

Titov³

2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

Modern sentence-level NMT systems often produce plausible translations of isolated sentences. However, when put in context, these translations may end up being inconsistent with each other. We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations. DocRepair performs automatic post-editing on a sequence of sentence-level translations, refining translations of sentences in context of each other. For training, the DocRepair model requires only monolingual document-level data in the target language. It is trained as a monolingual sequence-to-sequence model that maps inconsistent groups of sentences into consistent ones. The consistent groups come from the original training data; the inconsistent groups are obtained by sampling roundtrip translations for each isolated sentence. We show that this approach successfully imitates inconsistencies we aim to fix: using contrastive evaluation, we show large improvements in the translation of several contextual phenomena in an English→Russian translation task, as well as improvements in the BLEU score. We also conduct a human evaluation and show a strong preference of the annotators to corrected translations over the baseline ones. Moreover, we analyze which discourse phenomena are hard to capture using monolingual data only. 1 1 The code and data sets (including round-trip translations) are available at https://github.com/lena-voita/ good-translation-wrong-in-context.

show abstract

Section: Automatic Post-editingmentioning

confidence: 98%

Section: Automatic Post-editingmentioning

confidence: 99%

Context-Aware Monolingual Repair for Neural Machine Translation

Voita¹,

Sennrich²,

Titov³

2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

show abstract

“…Iterative Refinement has been studied in machine translation (Lee et al, 2018;Freitag et al, 2019;Mansimov et al, 2019;Kasai et al, 2020) to gradually improve translation quality. Refinement is also used with masked language models to improve fluency of non-autoregressive generation outputs (Ghazvininejad et al, 2019;Lawrence et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation

Hua¹,

Wang²

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Pre-trained Transformers have enabled impressive breakthroughs in generating long and fluent text, yet their outputs are often "rambling" without coherently arranged content.In this work, we present a novel content-controlled text generation framework, PAIR, with planning and iterative refinement, which is built upon a large model, BART. We first adapt the BERT model to automatically construct the content plans, consisting of keyphrase assignments and their corresponding sentence-level positions. The BART model is employed for generation without modifying its structure. We then propose a refinement algorithm to gradually enhance the generation quality within the sequence-tosequence framework. Evaluation with automatic metrics shows that adding planning consistently improves the generation quality on three distinct domains, with an average of 20 BLEU points and 12 METEOR points improvements. In addition, human judges rate our system outputs to be more relevant and coherent than comparisons without planning.

show abstract

“…They combined artificial training data with real data provided by WMT APE tasks to train their model. Using a similar approach of generating artificial APE data, Freitag et al (2019) trained a monolingual re-writing APE model trained on the generated artificial training data alone. Contrary to the round-trip translation approach, largescale artificial APE data was generated by simply translating source sentences using NMT and SMT systems and using the reference translations as the "pseudo" post-edits to create eSCAPE corpus .…”

Section: Related Workmentioning

confidence: 99%

Can Automatic Post-Editing Improve NMT?

Chollampatt

Susanto

Tan

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Automatic post-editing (APE) aims to improve machine translations, thereby reducing human post-editing effort. APE has had notable success when used with statistical machine translation (SMT) systems but has not been as successful over neural machine translation (NMT) systems. This has raised questions on the relevance of APE task in the current scenario. However, the training of APE models has been heavily reliant on large-scale artificial corpora combined with only limited human post-edited data. We hypothesize that APE models have been underperforming in improving NMT translations due to the lack of adequate supervision. To ascertain our hypothesis, we compile a larger corpus of human post-edits of English to German NMT. We empirically show that a state-of-art neural APE model trained on this corpus can significantly improve a strong in-domain NMT system, challenging the current understanding in the field. We further investigate the effects of varying training data sizes, using artificial training data, and domain specificity for the APE task. We release this new corpus under CC BY-NC-SA 4.0 license at https:// github.com/shamilcm/pedra.

show abstract

APE at Scale and Its Implications on MT Evaluation Biases

Cited by 48 publications

References 32 publications

Context-Aware Monolingual Repair for Neural Machine Translation

Context-Aware Monolingual Repair for Neural Machine Translation

PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation

Can Automatic Post-Editing Improve NMT?

Contact Info

Product

Resources

About