Multilingual Neural Machine Translation with Language Clustering

Abstract

Multilingual neural machine translation (NMT), which translates multiple languages using a single model, is of great practical importance due to its advantages in simplifying the training process, reducing online maintenance costs, and enhancing low-resource and zero-shot translation. Given there are thousands of languages in the world and some of them are very different, it is extremely burdensome to handle them all in a single model or use a separate model for each language pair. Therefore, given a fixed resource budget, e.g., the number of models, how to determine which languages should be supported by one model is critical to multilingual NMT, which, unfortunately, has been ignored by previous work. In this work, we develop a framework that clusters languages into different groups and trains one multilingual model for each cluster. We study two methods for language clustering: (1) using prior knowledge, where we cluster languages according to language family, and (2) using language embedding, in which we represent each language by an embedding vector and cluster them in the embedding space. In particular, we obtain the embedding vectors of all the languages by training a universal neural machine translation model. Our experiments on 23 languages show that the first clustering method is simple and easy to understand but leading to suboptimal translation accuracy, while the second method sufficiently captures the relationship among languages well and improves the translation accuracy for almost all the languages over baseline methods.

Investigating Multilingual NMT Representations at Scale

“…Our conclusions are similar to that of works that have attempted to cluster learned language vectors:Östling and Tiedemann (2016); Tan et al (2019) both find that hierarchical clusters of language vectors discover linguistic similarity, with the former finding fine-grained clusterings for Germanic languages. In a similar vein, Tiedemann (2018) visualizes language vectors and find that they roughly cluster by linguistic family.…”

“…Recent work on interpretability for NLU tasks uses methods such as diagnostic tasks (Belinkov et al, 2017;Tenney et al, 2019;Belinkov et al, 2018), attention based methods (Raganato and Tiedemann, 2018) or task analysis (Zhang and Bowman, 2018) and is primarily focused on understanding the linguistic features encoded by a trained model. Some recent works compare learned language vectors (Östling and Tiedemann, 2016;Tan et al, 2019;Tiedemann, 2018) and find conclusions similar to ours. To the best of our knowledge, we are the first to compare the hidden representations of the sentences themselves.…”

A Survey of Multilingual Neural Machine Translation

“…We expect that the balance between language-agnostic and language-specific representations should depend on the language pairs. Prasanna [117], Tan et al [141] are some of the works that cluster languages into language families and train separate MNMT models per family. Language families can be decided by using linguistic knowledge 7 [117] or by using embedding similarities where the embeddings are obtained from a multilingual word2vec model [141].…”

BAM! Born-Again Multi-Task Networks for Natural Language Understanding

“…In reinforcement learning, knowledge distillation has been used to regularize multi-task agents (Parisotto et al, 2016;Teh et al, 2017). In NLP, Tan et al (2019) distill singlelanguage-pair machine translation systems into a many-language system. However, they focus on multilingual rather than multi-task learning, use a more complex training procedure, and only experiment with Single→Multi distillation.…”