Findings of the WMT 2018 Shared Task on Automatic Post-Editing

Chatterjee, Rajen; Negri, Matteo; Rubino, Raphaël; Turchi, Marco

doi:10.18653/v1/w18-6452

Cited by 27 publications

(39 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The repetition rate measures the repetitiveness inside a text by looking at the rate of non-singleton n-gram types (n=1...4) and combining them using the geometric mean. Larger values indicate a higher text repetitiveness and, as discussed in (Bojar et al, 2016;Bojar et al, 2017;Chatterjee et al, 2018a), suggest a higher chance of learning from the training set correction patterns that are applicable also to the test set. In the previous rounds of the task, we considered the large differences in repetitiveness across the datasets as a possible explanation for the variable gains over the baseline obtained by participants.…”

Section: Complexity Indicators: Repetition Ratementioning

confidence: 96%

“…As discussed in (Bojar et al, 2017;Chatterjee et al, 2018a), numeric evidence of a higher quality of the original translations can indicate a smaller room for improvement for APE systems (having, at the same time, less to learn during training and less to correct at test stage). On one side, indeed, training on good (or near-perfect) automatic translations can drastically reduce the number of learned correction patterns.…”

Section: Complexity Indicators: Mt Qualitymentioning

confidence: 99%

See 1 more Smart Citation

Findings of the WMT 2019 Shared Task on Automatic Post-Editing

Chatterjee¹,

Federmann²,

Negri³

et al. 2019

Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)

Self Cite

View full text Add to dashboard Cite

We present the results from the 5 th round of the WMT task on MT Automatic Post-Editing. The task consists in automatically correcting the output of a "black-box" machine translation system by learning from human corrections. Keeping the same general evaluation setting of the previous four rounds, this year we focused on two language pairs (English-German and English-Russian) and on domain-specific data (Information Technology). For both the language directions, MT outputs were produced by neural systems unknown to participants. Seven teams participated in the English-German task, with a total of 18 submitted runs. The evaluation, which was performed on the same test set used for the 2018 round, shows further progress in APE technology: 4 teams achieved better results than last year's winning system, with improvements up to-0.78 TER and +1.23 BLEU points over the baseline. Two teams participated in the English-Russian task submitting 2 runs each. On this new language direction, characterized by a higher quality of the original translations, the task proved to be particularly challenging. Indeed, none of the submitted runs improved the very high results of the strong system used to produce the initial translations (16.16 TER, 76.20 BLEU).

show abstract

Section: Complexity Indicators: Repetition Ratementioning

confidence: 96%

Section: Complexity Indicators: Mt Qualitymentioning

confidence: 99%

Findings of the WMT 2019 Shared Task on Automatic Post-Editing

Chatterjee¹,

Federmann²,

Negri³

et al. 2019

Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, developing a human-edit corpus is time-consuming and costly, so it cannot be performed for all language pairs. There is not a clear performance improvement when NMT results are post-edited using neural networks, unlike when SMT results are post-edited using neural networks [17].…”

Section: Related Workmentioning

confidence: 94%

Hybrid Translation with Classification: Revisiting Rule-Based and Neural Machine Translation

2020

View full text Add to dashboard Cite

This paper proposes a hybrid machine-translation system that combines neural machine translation with well-developed rule-based machine translation to utilize the stability of the latter to compensate for the inadequacy of neural machine translation in rare-resource domains. A classifier is introduced to predict which translation from the two systems is more reliable. We explore a set of features that reflect the reliability of translation and its process, and training data is automatically expanded with a small, human-labeled dataset to solve the insufficient-data problem. A series of experiments shows that the hybrid system’s translation accuracy is improved, especially in out-of-domain translations, and classification accuracy is greatly improved when using the proposed features and the automatically constructed training set. A comparison between feature- and text-based classification is also performed, and the results show that the feature-based model achieves better classification accuracy, even when compared to neural network text classifiers.

show abstract

“…Exploiting synthetic data has shown to lead to improvements in APE (Bojar et al 2017;Chatterjee et al 2018) and in NMT systems (Sennrich et al 2016b;Poncelas et al 2018). A detailed summary is presented in do Carmo et al (2020).…”

Section: Data Demands and Data Scarcitymentioning

confidence: 99%

A roadmap to neural automatic post-editing: an empirical approach

Shterionov

Carmo

Moorkens

et al. 2020

Machine Translation

View full text Add to dashboard Cite

In a translation workflow, machine translation (MT) is almost always followed by a human post-editing step, where the raw MT output is corrected to meet required quality standards. To reduce the number of errors human translators need to correct, automatic post-editing (APE) methods have been developed and deployed in such workflows. With the advances in deep learning, neural APE (NPE) systems have outranked more traditional, statistical, ones. However, the plethora of options, variables and settings, as well as the relation between NPE performance and train/test data makes it difficult to select the most suitable approach for a given use case. In this article, we systematically analyse these different parameters with respect to NPE performance. We build an NPE “roadmap” to trace the different decision points and train a set of systems selecting different options through the roadmap. We also propose a novel approach for APE with data augmentation. We then analyse the performance of 15 of these systems and identify the best ones. In fact, the best systems are the ones that follow the newly-proposed method. The work presented in this article follows from a collaborative project between Microsoft and the ADAPT centre. The data provided by Microsoft originates from phrase-based statistical MT (PBSMT) systems employed in production. All tested NPE systems significantly increase the translation quality, proving the effectiveness of neural post-editing in the context of a commercial translation workflow that leverages PBSMT.

show abstract

Findings of the WMT 2018 Shared Task on Automatic Post-Editing

Cited by 27 publications

References 26 publications

Findings of the WMT 2019 Shared Task on Automatic Post-Editing

Findings of the WMT 2019 Shared Task on Automatic Post-Editing

Hybrid Translation with Classification: Revisiting Rule-Based and Neural Machine Translation

A roadmap to neural automatic post-editing: an empirical approach

Contact Info

Product

Resources

About