Multilingual Neural Machine Translation for Low-Resource Languages

Abstract

In recent years, Neural Machine Translation (NMT) has been shown to be more effective than phrase-based statistical methods, thus quickly becoming the state of the art in machine translation (MT). However, NMT systems are limited in translating low-resourced languages, due to the significant amount of parallel data that is required to learn useful mappings between languages. In this work, we show how the so-called multilingual NMT can help to tackle the challenges associated with low-resourced language translation. The underlying principle of multilingual NMT is to force the creation of hidden representations of words in a shared semantic space across multiple languages, thus enabling a positive parameter transfer across languages. Along this direction, we present multilingual translation experiments with three languages (English, Italian, Romanian) covering six translation directions, utilizing both recurrent neural networks and transformer (or self-attentive) neural networks. We then focus on the zero-shot translation problem, that is how to leverage multi-lingual data in order to learn translation directions that are not covered by the available training material. To this aim, we introduce our recently proposed iterative self-training method, which incrementally improves a multilingual NMT on a zero-shot direction by just relying on monolingual data. Our results on TED talks data show that multilingual NMT outperforms conventional bilingual NMT, that the transformer NMT outperforms recurrent NMT, and that zero-shot NMT outperforms conventional pivoting methods and even matches the performance of a fully-trained bilingual system.

Improving Zero-shot Multilingual Neural Machine Translation for Low-Resource Languages

“…Moreover, the attention mechanism of the many-toone strategy is fine-tuned by the generated pseudo parallel corpus. Following google's multilingual NMT, Lakew et al (2017) proposed the self-learning algorithm for zero-shot NMT, which is a process of constantly iterating through a train-infer-train mechanism to generate synthetic parallel data for zero-shot translation. By using this way, the quality of the generated parallel data is significantly improved.…”

“…Particularly by using this way, the multilingual NMT system has possibility to perform zero-shot translation through sharing the common model. In order to improve the performance of zero-shot languages, Lakew et al (2017) proposed the selflearning algorithm, which generates synthetic parallel data by translating existing target data through the multilingual NMT round and round. The whole process is a self-learning cycle of train-infer-train.…”

A Joint Back-Translation and Transfer Learning Method for Low-Resource Neural Machine Translation

“…Inspired by multilingual NMT, Gu et al improved translation quality for tiny even zero-resource parallel corpus by sharing a universal word-level representation and sentence-level representation [34]. For zero-resource parallel corpus, Lakew et al generated new synthetic data by the traininginference-training scheme, which is based on a multi-NMT system [35]. Xia et al proposed a dual-learning algorithm called dual-NMT to tackle the training data bottleneck, which teach each other by giving feedback signals [36].…”

An Empirical Investigation of Multi-bridge Multilingual NMT models

“…Particularly, multilingual NMT models using English-centric parallel corpora have shown significant improvements for translation between English and low-resources languages (Firat, Cho, and Bengio 2016;Johnson et al 2017). Translation between non-English languages has received lesser attention, with the default approach being pivot translation (Lakew et al 2017). Pivot translation is a strong baseline, but needs multiple decoding steps resulting in increased latency and cascading errors.…”

“…Hence, vanilla zeroshot translation quality significantly lags behind pivot translation. Various methods have been proposed to address these limitations by aligning encoder representations (Arivazhagan et al 2019) or using pseudo-parallel corpus between non-English languages during training (Lakew et al 2017).…”