Multilingual NMT with a Language-Independent Attention Bridge

Abstract

In this paper, we propose a multilingual encoder-decoder architecture capable of obtaining multilingual sentence representations by means of incorporating an intermediate attention bridge that is shared across all languages. That is, we train the model with language-specific encoders and decoders that are connected via self-attention with a shared layer that we call attention bridge. This layer exploits the semantics from each language for performing translation and develops into a language-independent meaning representation that can efficiently be used for transfer learning. We present a new framework for the efficient development of multilingual NMT using this model and scheduled training. We have tested the approach in a systematic way with a multi-parallel data set. We show that the model achieves substantial improvements over strong bilingual models and that it also works well for zero-shot translation, which demonstrates its ability of abstraction and transfer learning.

A Survey of Multilingual Neural Machine Translation

“…Hence, the decoder's attention mechanism sees a variable number of encoder representations for equivalent sentences across languages. To overcome this, an attention bridge network generates a fixed number of contextual representations that are input to the attention network [94,149]. By minimizing the diversity of representations, the decoder's task is simplified and it becomes better at language generation.…”

Language-aware Interlingua for Multilingual Neural Machine Translation

“…the i-th column of I emb represents a initial semantic subspace that guides what semantic information of the H enc should be attended to at the corresponding position i of the Interlingua output. The r means every Encoder H enc will be mapped into a fixed size representation of r hidden states, and it is set to 10 during all of our experiments, similar to the work of (Vázquez et al, 2018). By incorporating a shared interlingua embedding, we expect that it can exploit the semantics of various subspaces from encoded representation, and the same semantic components of different sentences from both same and different languages should be mapped into the same posi-…”

“…During all our experiments, we follow the settings of TRANS-FORMER-base (Vaswani et al, 2017) with hidden/embedding size 512, 6 hidden layers and 8 attention heads. We set 3 layers for Interlingua, and r = 10 similar to the work of (Vázquez et al, 2018). We apply sub-word NMT (Sennrich et al, 2015), where a joint BPE model is trained for all languages with 50,000 operations.…”

“…In regards to generating language-independent representation, Lu et al (2018) and Vázquez et al (2018) both attempted to build a similar language-independent representation. However, their work is all based on multiple languagedependent LSTM Encoder-Decoders, which significantly increase the model complexity.…”

A Multilingual Neural Machine Translation Model for Biomedical Data

“…The model-centric approaches, on the other hand, center on adjusting the training objectives (Wang et al, 2017;Tan et al, 2019b); modifying the model architectures (Vázquez et al, 2019;Dou et al, 2019a); and tweaking the decoding procedure (Hasler et al, 2018;Dou et al, 2019b).…”