A Systematic Study of Inner-Attention-Based Sentence Representations in Multilingual Neural Machine Translation

Abstract

<jats:p> Neural machine translation has considerably improved the quality of automatic translations by learning good representations of input sentences. In this article, we explore a multilingual translation model capable of producing fixed-size sentence representations by incorporating an intermediate crosslingual shared layer, which we refer to as attention bridge. This layer exploits the semantics from each language and develops into a language-agnostic meaning representation that can be efficiently used for transfer learning. We systematically study the impact of the size of the attention bridge and the effect of including additional languages in the model. In contrast to related previous work, we demonstrate that there is no conflict between translation performance and the use of sentence representations in downstream tasks. In particular, we show that larger intermediate layers not only improve translation quality, especially for long sentences, but also push the accuracy of trainable classification tasks. Nevertheless, shorter representations lead to increased compression that is beneficial in non-trainable similarity tasks. Similarly, we show that trainable downstream tasks benefit from multilingual models, whereas additional language signals do not improve performance in non-trainable benchmarks. This is an important insight that helps to properly design models for specific applications. Finally, we also include an in-depth analysis of the proposed attention bridge and its ability to encode linguistic properties. We carefully analyze the information that is captured by individual attention heads and identify interesting patterns that explain the performance of specific settings in linguistic probing tasks. </jats:p>

Proceedings of the 17th International Conference on Spoken Language Translation

“…HY (Vázquez et al, 2020) participated with 14 http://sox.sourceforge.net/ both cascade and end-to-end systems. For the end-to-end system, they used a multimodal approach (with audio and text as the two modalities treated as different languages) trained in a multitask fashion, which maps the internal representations of different encoders into a shared space before decoding.…”

“…For the end-to-end system, they used a multimodal approach (with audio and text as the two modalities treated as different languages) trained in a multitask fashion, which maps the internal representations of different encoders into a shared space before decoding. To this aim, they incorporated the inner-attention based architecture proposed by (Vázquez et al, 2020) within Transformer-based encoders (inspired by (Tu et al, 2019;Di Gangi et al, 2019c)) and decoders. For the cascade approach, they used a pipeline of three stages: (1) ASR (trained with S-Transformer (Di Gangi et al, 2019c)), (2) re-punctuation and letter case restoration (based on Marian's implementation of Transformer), and ( 3) MT (also based on Marian).…”

“…End-to-end SLT We use an inner-attention based architecture proposed by Vázquez et al (2020). In a nutshell, it follows the conventional structure of an encoderdecoder model of MT Luong et al, 2016) enabled with multilingual training by incorporating language-specific encoders and decoders trainable with a language-rotating scheduler (Dong et al, 2015;Schwenk and Douze, 2017), and an intermediate shared inner-attention layer (Cífka and Bojar, 2018;Lu et al, 2018).…”

“…We focus on exploiting the knowledge-transfer capabilities of a multitasking architecture based on language-specific encoders-decoders (Lu et al, 2018;Schwenk and Douze, 2017;Luong et al, 2016). This idea has been proposed and studied in the multilingual scenario (Vázquez et al, 2020;Subramanian et al, 2018;Firat et al, 2017), however, we adapt it to be used in a multimodal scenario. Regarding different modalities (in this case, audio and text) as different languages when training the model, allows us to employ a cross-modal intermediate shared layer for performing SLT in an end-to-end fashion.…”

Controlling the Imprint of Passivization and Negation in Contextualized Representations

“…We propose to explore the impact of context variation on word representations. We specifically address representations generated by the BERT model (Devlin et al, 2019), trained using a language modeling objective, and translation models involving one or more language pairs (Artetxe and Schwenk, 2019;Vázquez et al, 2020).…”