Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation

Abstract

This paper demonstrates that multilingual pretraining, a proper fine-tuning method and a large-scale parallel dataset from multiple auxiliary languages are all critical for zeroshot translation, where the NMT model is tested on source languages unseen during supervised training. Following this idea, we present SixT++, a strong many-to-English NMT model that supports 100 source languages but is trained once with a parallel dataset from only six source languages. SixT++ initializes the decoder embedding and the full encoder with XLM-R large, and then trains the encoder and decoder layers with a simple two-stage training strategy. SixT++ achieves impressive performance on many-to-English translation. It significantly outperforms CRISS and m2m-100, two strong multilingual NMT systems, with an average gain of 7.2 and 5.0 BLEU respectively. Additionally, SixT++ offers a set of model parameters that can be further fine-tuned to develop unsupervised NMT models for low-resource languages. With back-translation on monolingual data of low-resource language, it outperforms all current state-of-the-art unsupervised methods on Nepali and Sinhal for both translating into and from English.

Multilingual Bidirectional Unsupervised Translation through Multilingual Finetuning and Back-Translation

“…This is possible, but our work explores a language-agnostic multilingual setting. We refer the interested reader to (Zhang et al, 2022), which finds through an empirical study that overall multilingual translation performance is best when languages are balanced.…”

Refining Low-Resource Unsupervised Translation by Language Disentanglement of Multilingual Model

“…We evaluate our method on the FLoRes [13] low-resource unsupervised translation tasks on English (En) to and from Nepali (Ne), Sinhala (Si) and Gujarati (Gu) from the Indic family, as well as other low-resource languages: Latvian (Lv) and Estonian (Et) from the Uralic family and Kazakh (Kk) from the Turkic family. Although Hindi (Hi) and Finnish (Fi) are relatively high-resourced compared to their respective siblings, we still add them into the mix for Indic and Uralic families respectively to help the learning process of their respective low-resource siblings, following [12,8].…”

“…Training the model in such a multilingual environment helps the encoder to learn language-agnostic latent representations that are shared across multiple languages, allowing the decoder to translate from any language. The vast availability of high-resource siblings (e.g., Hindi for Indic languages) may also help improve the performance of low-resource MT significantly [12,8].…”

“…Garcia et al [11,12] propose multi-stage training procedures that combine both monolingual data and auxiliary parallel data to improve the performance of low-resource translation. Meanwhile, Chen et al [7,8] suggest to scale their experiments up to 100 languages to attain better performance boosts. Nonetheless, while such line of work offers considerable insight into low-resource MT, it is not within the scope of this paper, as we focus on fully unsupervised machine translation where absolutely no parallel data is available, regardless of languages.…”

Improving neural machine translation: data centric approaches

“…Training the model in such a multilingual environment helps the encoder learn language-agnostic latent representations that are shared across multiple languages, allowing the decoder to translate from any language. The vast availability of high-resource siblings (e.g., Hindi for Indic languages) may help improve the performance of low-resource MT significantly (Garcia et al, 2021, Chen et al, 2021a.…”