Multi-source transformer with combined losses for automatic post editing

Abstract

Recent approaches to the Automatic Postediting (APE) of Machine Translation (MT) have shown that best results are obtained by neural multi-source models that correct the raw MT output by also considering information from the corresponding source sentence. To this aim, we present for the first time a neural multi-source APE model based on the Transformer architecture. Moreover, we employ sequence-level loss functions in order to avoid exposure bias during training and to be consistent with the automatic evaluation metrics used for the task. These are the main features of our submissions to the WMT 2018 APE shared task, where we participated both in the PBSMT subtask (i.e. the correction of MT outputs from a phrase-based system) and in the NMT subtask (i.e. the correction of neural outputs). In the first subtask, our system improves over the baseline up to-5.3 TER and +8.23 BLEU points ranking second out of 11 submitted runs. In the second one, characterized by the higher quality of the initial translations, we report lower but statistically significant gains (up to-0.38 TER and +0.8 BLEU), ranking first out of 10 submissions.

Findings of the WMT 2019 Shared Task on Automatic Post-Editing

“…The other approach exploits a retrieval-based method similar to (Farajian et al, 2017): given a query containing the source and the MT output to be post-edited, it: i) retrieves similar triplets from the training data, ii) ranks them based on the sentence level BLEU score between the MT output and the post-edit, and iii) creates the token based on the TER computed between the MT output and the post-edit of the most similar triplet. The backbone architecture is the multi-source extension of Transformer (Vaswani et al, 2017) described in (Tebbifakhr et al, 2018), which is trained both on the task data and on the available artificial corpora.…”

Deep Copycat Networks for Text-to-Text Generation

“…SOTA APE approaches tackle the task with the multi-source transformer architectures (Junczys-Dowmunt and Grundkiewicz, 2018;Tebbifakhr et al, 2018). Two encoders encode the source and the raw MT, respectively, and an additional targetsource multi-head attention component is stacked on top of the original target-source multi-head attention component.…”

“…7 We see that the performance of our copycat systems depends on the difficulty of the task: we observe an improvement for Latvian (+1 absolute HTER point improvement) with the lowest baseline quality (HTER= 0.29), and minor improvements for German with a higher baseline quality (on average 0.14 HTER). As a comparison, the best performing SOTA systems achieve only up to 0.4 HTER improvement for the German dataset and this is by using millions of training data, combined losses and ensembling techniques, whereas we use minimum external resources (Tebbifakhr et al, 2018;Junczys-Dowmunt and Grundkiewicz, 2018). 8 For EN-DE, our PNT-TRG model performs a small number of corrections (mostly accurate) to the outputs: only 50 sentences, compared to the 200-300 sentences modified by the SOTA EN-DE APE systems .…”

USAAR-DFKI – The Transference Architecture for English–German Automatic Post-Editing

“…They introduce an additional joint encoder that attends over a combination of the two encoded sequences from mt and src. Tebbifakhr et al (2018), the NMT-subtask winner of WMT 2018 (wmt18 nmt best ), employ sequence-level loss functions in order to avoid exposure bias during training and to be consistent with the automatic evaluation metrics. Shin and Lee (2018) propose that each encoder has its own self-attention and feed-forward layer to process each input separately.…”