Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation

Abstract

This paper describes the Microsoft Translator submissions to the WMT19 news translation shared task for English-German. Our main focus is document-level neural machine translation with deep transformer models. We start with strong sentence-level baselines, trained on large-scale data created via data-filtering and noisy back-translation and find that backtranslation seems to mainly help with translationese input. We explore fine-tuning techniques, deeper models and different ensembling strategies to counter these effects. Using document boundaries present in the authentic and synthetic parallel data, we create sequences of up to 1000 subword segments and train transformer translation models. We experiment with data augmentation techniques for the smaller authentic data with documentboundaries and for larger authentic data without boundaries. We further explore multi-task training for the incorporation of documentlevel source language monolingual data via the BERT-objective on the encoder and twopass decoding for combinations of sentencelevel and document-level systems. Based on preliminary human evaluation results, evaluators strongly prefer the document-level systems over our comparable sentence-level system. The document-level systems also seem to score higher than the human references in source-based direct assessment.

Analysing concatenation approaches to document-level NMT in two different domains

“…For the non-sliding window approaches with fixed maximum size, sentence splitting is not as straightforward and requires some additional treatment. Segments are also separated by separation tokens but we realized that they do not necessarily match with the segment boundaries in the reference data even though the original paper suggests that this should be rather stable (Junczys-Dowmunt, 2019). This is especially fatal if the number of segments does not match.…”

“…• we train different context-aware MT models (Tiedemann and Scherrer, 2017;Agrawal et al, 2018;Maruf et al, 2019;Junczys-Dowmunt, 2019) on the two datasets and evaluate them using a comparative setup with artificially scrambled data.…”

Meta-Learning for Few-Shot NMT Adaptation

“…We use WMT14 and WMT19 newtests as validation and test sets respectively. The baseline system scores 37.99 BLEU on the full WMT19 newstest which compares favorably with strong single system baselines at WMT19 shared task Junczys-Dowmunt, 2019).…”

Analysing Coreference in Transformer Outputs

“…Document-level machine translation (MT) systems are needed to deal with coreference as a whole. Although some attempts to include extrasentential information exist (Wang et al, 2017;Voita et al, 2018;Jean and Cho, 2019;Junczys-Dowmunt, 2019), the problem is far from being solved. Besides that, some further problems of NMT that do not seem to be related to coreference at first glance (such as translation of unknown words and proper names or the hallucination of additional words) cause coreference-related errors.…”