An Empirical Analysis of NMT-Derived Interlingual Embeddings and Their Use in Parallel Sentence Identification

Abstract

End-to-end neural machine translation has overtaken statistical machine translation in terms of translation quality for some language pairs, specially those with large amounts of parallel data. Besides this palpable improvement, neural networks provide several new properties. A single system can be trained to translate between many languages at almost no additional cost other than training time. Furthermore, internal representations learned by the network serve as a new semantic representation of words or sentences which, unlike standard word embeddings, are learned in an essentially bilingual or even multilingual context. In view of these properties, the contribution of the present work is two-fold. First, we systematically study the NMT context vectors, i.e. output of the encoder, and their power as an interlingua representation of a sentence. We assess their quality and effectiveness by measuring similarities across translations, as well as semantically related and semantically unrelated sentence pairs. Second, as extrinsic evaluation of the first point, we identify parallel sentences in comparable corpora, obtaining an F 1 = 98.2% on data from a shared task when using only NMT context vectors. Using context vectors jointly with similarity measures F 1 reaches 98.9%.

Identifying Semantic Divergences in Parallel Text without Annotations

“…Similar features have proved to be useful for the related task of translation quality estimation (Specia et al, 2010(Specia et al, , 2016, which aims to detect divergences introduced by MT errors, rather than human translation. Recently, sentence embeddings have also been used to detect parallelism (España-Bonet et al, 2017;Schwenk and Douze, 2017). Although embeddings capture semantic generalizations, these models are trained with neural MT objectives, which do not distinguish semantically equivalent segments from divergent parallel segments.…”

Simple and Effective Paraphrastic Similarity from Parallel Translations

“…This task consists of finding the gold aligned parallel sentences given two large corpora in two distinct languages. Typically, only 8 Including a 3-layer transformer trained on a constructed parallel corpus (Chidambaram et al, 2018), a bidirectional gated recurrent unit (GRU) network trained on a collection of parallel corpora using en-es, en-ar, and ar-es bitext (Espana-Bonet et al, 2017), and a 3 layer bidirectional LSTM trained on 9 languages in Europarl (Schwenk, 2018 Table 3 on the French and German mining tasks demonstrate the proposed model outperforms Schwenk (2018), although the gap is substantially smaller than on the STS tasks. The reason for this is likely the domain mismatch between the STS data (image captions) and the training data (Europarl).…”

XCMRC: Evaluating Cross-Lingual Machine Reading Comprehension

“…Zhou (2016) learned bilingual document representations by minimizing the Euclidean distance between document representations and their translations [30]. Conneau (2017) and Espa (2017) jointly trained a sequence to sequence MT system on multiple languages to learn a shared multilingual sentence embedding space [4] [9]. Our method leverages the latest breakthrough in NLP: BERT [7] as the multilingual sentence encoder.…”