Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder

Ha, Thanh-Le; Niehues, Jan; Waibel, Alexander

doi:10.48550/arxiv.1611.04798

Cited by 152 publications

(97 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multilingual modeling has been an important topic in applying sequence-to-squence models to language applications ranging from machine translation [1,2] to automatic speech recognition (ASR) [3]. It is possible to employ one single neural model for multiple datasets with different languages with the goal of capturing the shared features between the languages.…”

Section: Introductionmentioning

confidence: 99%

Efficient Weight factorization for Multilingual Speech Recognition

Pham¹,

Nguyen²,

Stueker³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

End-to-end multilingual speech recognition involves using a single model training on a compositional speech corpus including many languages, resulting in a single neural network to handle transcribing different languages. Due to the fact that each language in the training data has different characteristics, the shared network may struggle to optimize for all various languages simultaneously. In this paper we propose a novel multilingual architecture that targets the core operation in neural networks: linear transformation functions. The key idea of the method is to assign fast weight matrices for each language by decomposing each weight matrix into a shared component and a language dependent component. The latter is then factorized into vectors using rank-1 assumptions to reduce the number of parameters per language. This efficient factorization scheme is proved to be effective in two multilingual settings with 7 and 27 languages, reducing the word error rates by 26% and 27% rel. for two popular architectures LSTM and Transformer, respectively.

show abstract

Section: Introductionmentioning

confidence: 99%

Efficient Weight factorization for Multilingual Speech Recognition

Pham¹,

Nguyen²,

Stueker³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Multilingual NMT Multilingual NMT aims to train one model to serve all language pairs (Ha et al, 2016;Firat et al, 2016;Johnson et al, 2017). Several subsequent works explored various parameter sharing strategies to mitigate the representation bottleneck (Blackwood et al, 2018;Platanios et al, 2018;Sen et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

Improving Multilingual Translation by Representation and Gradient Regularization

Yang

Eriguchi

Muzio

et al. 2021

Preprint

View full text Add to dashboard Cite

Multilingual Neural Machine Translation (NMT) enables one model to serve all translation directions, including ones that are unseen during training, i.e. zero-shot translation. Despite being theoretically attractive, current models often produce low quality translations -commonly failing to even produce outputs in the right target language. In this work, we observe that off-target translation is dominant even in strong multilingual systems, trained on massive multilingual corpora. To address this issue, we propose a joint approach to regularize NMT models at both representation-level and gradient-level. At the representation level, we leverage an auxiliary target language prediction task to regularize decoder outputs to retain information about the target language. At the gradient level, we leverage a small amount of direct data (in thousands of sentence pairs) to regularize model gradients. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance by +5.59 and +10.38 BLEU on WMT and OPUS datasets respectively. Moreover, experiments show that our method also works well when the small amount of direct data is not available. 1 * Work done while interning at Microsoft.

show abstract

“…But, due to the high complexity of the previously mentioned method, Johnson et al and Ha et al attempted to build a multilingual NMT without modifying the network architecture. Ha, Niehues, and Waibel (2016) applied a languagespecific coding to words of both source and target languages for better representation. In practice, language-specific coding for words and sub-words significantly increased the length of sentences, which causes trouble for sentence representation and attention mechanism.…”

Section: Multilingual Nmtmentioning

confidence: 99%

“…Because of the fact that NMT system is highly depended on extensive high-quality parallel data, which can only be acquired for few language pairs, it is still challenging for low-resource and zero-shot NMT (Koehn and Knowles 2017). Existing approaches for zero-shot NMT include multilingual NMT Ha, Niehues, and Waibel 2016;Johnson et al 2017), interactive multimodal framework (Kiros, Salakhutdinov, and Zemel 2014;Nakayama and Nishida 2017), pivot-based NMT (Wu and Wang 2007;Cheng et al 2016;Leng et al 2019) and teacher-student architecture (Chen et al 2017).…”

Section: Introductionmentioning

confidence: 99%

Improving Zero-shot Multilingual Neural Machine Translation for Low-Resource Languages

Li,

Luo

2021

Preprint

View full text Add to dashboard Cite

Although the multilingual Neural Machine Translation(NMT), which extends Google's multilingual NMT, has ability to perform zero-shot translation and the iterative self-learning algorithm can improve the quality of zero-shot translation, it confronts with two problems: the multilingual NMT model is prone to generate wrong target language when implementing zero-shot translation; the self-learning algorithm, which uses beam search to generate synthetic parallel data, demolishes the diversity of the generated source language and amplifies the impact of the same noise during the iterative learning process. In this paper, we propose the tagged-multilingual NMT model and improve the self-learning algorithm to handle these two problems. Firstly, we extend the Google's multilingual NMT model and add target tokens to the target languages, which associates the start tag with the target language to ensure that the source language can be translated to the required target language. Secondly, we improve the self-learning algorithm by replacing beam search with random sample to increases the diversity of the generated data and makes it properly cover the true data distribution. Experimental results on IWSLT show that the adjusted tagged-multilingual NMT separately obtains 9.41 and 7.85 BLEU scores over the multilingual NMT on 2010 and 2017 Romanian-Italian test sets. Similarly, it obtains 9.08 and 7.99 BLEU scores on Italian-Romanian zero-shot translation. Furthermore, the improved self-learning algorithm shows its superiorities over the conventional self-learning algorithm on zero-shot translations.

show abstract

Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder

Cited by 152 publications

References 8 publications

Efficient Weight factorization for Multilingual Speech Recognition

Efficient Weight factorization for Multilingual Speech Recognition

Improving Multilingual Translation by Representation and Gradient Regularization

Improving Zero-shot Multilingual Neural Machine Translation for Low-Resource Languages

Contact Info

Product

Resources

About