Consistency by Agreement in Zero-Shot Neural Machine Translation

Abstract

Generalization and reliability of multilingual translation often highly depend on the amount of available parallel data for each language pair of interest. In this paper, we focus on zero-shot generalization-a challenging setup that tests models on translation directions they have not been optimized for at training time.To solve the problem, we (i) reformulate multilingual translation as probabilistic inference, (ii) define the notion of zero-shot consistency and show why standard training often results in models unsuitable for zero-shot tasks, and (iii) introduce a consistent agreement-based training method that encourages the model to produce equivalent translations of parallel sentences in auxiliary languages. We test our multilingual NMT models on multiple public zeroshot translation benchmarks (IWSLT17, UN corpus, Europarl) and show that agreementbased learning often results in 2-3 BLEU zeroshot improvement over strong baselines without any loss in performance on supervised translation directions. Philip Koehn. 2017. Europarl: A parallel corpus for statistical machine translation. Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. arXiv preprint arXiv:1706.03872. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043.

Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation

“…and enable zero-shot translation (i.e. direct translation between a language pair never seen in training) (Firat et al, 2016b;Johnson et al, 2017;Al-Shedivat and Parikh, 2019;Gu et al, 2019). Despite these potential benefits, multilingual NMT tends to underperform its bilingual counterparts (Johnson et al, 2017;Arivazhagan et al, 2019b) and results in considerably worse translation performance when many languages are accommodated (Aharoni et al, 2019).…”

Reference Language based Unsupervised Neural Machine Translation

“…Arivazhagan et al (2019) addressed the zeroshot generalization problem that some translation directions have not been optimized well due to a lack of parallel data. Al-Shedivat and Parikh (2019) introduced a consistent agreement-based training method that encourages the model to produce equivalent translations of parallel sentences in zero-shot translation, which share similarities with our RAT approach. However, in terms of a specific implementation, because of the differences between UNMT and NMT, we have provided three new UNMT methods, and have alleviated the problem of uncontrollable intermediate BT quality in UNMT.…”

Discrete Latent Variable Representations for Low-Resource Text Classification

“…In the binary latent variable setting, straight-through estimators are often used (Dong et al, 2019). Another choice is "continuous decoding" which takes a convex combination of latent values to make the loss differentiable (Al-Shedivat and Parikh, 2019). Yet a less considered choice is Hard EM (Brown et al, 1993;De Marcken, 1995;Spitkovsky et al, 2010).…”