Bilingual Distributed Word Representations from Document-Aligned Comparable Data

Abstract

We propose a new model for learning bilingual word representations from non-parallel document-aligned data. Following the recent advances in word representation learning, our model learns dense real-valued word vectors, that is, bilingual word embeddings (BWEs). Unlike prior work on inducing BWEs which heavily relied on parallel sentence-aligned corpora and/or readily available translation resources such as dictionaries, the article reveals that BWEs may be learned solely on the basis of document-aligned comparable data without any additional lexical resources nor syntactic information. We present a comparison of our approach with previous state-of-the-art models for learning bilingual word representations from comparable data that rely on the framework of multilingual probabilistic topic modeling (MuPTM), as well as with distributional local context-counting models. We demonstrate the utility of the induced BWEs in two semantic tasks: (1) bilingual lexicon extraction, (2) suggesting word translations in context for polysemous words. Our simple yet effective BWE-based models significantly outperform the MuPTM-based and contextcounting representation models from comparable data as well as prior BWE-based models, and acquire the best reported results on both tasks for all three tested language pairs.

Bilingual Lexicon Induction by Learning to Combine Word-Level and Character-Level Representations

“…The first variant of the architecture assumes that W S and W T are obtained in advance using any state-of-theart word embedding model, e.g., (Mikolov et al, 2013b;Vulić and Moens, 2016). They are then kept fixed when minimizing the loss from Eq.…”

“…To test the generality of our approach, we experiment with two well-known embedding models: (1) the model from Mikolov et al (2013b), which trains monolingual embeddings using skipgram with negative sampling (SGNS) (Mikolov et al, 2013a); and (2) the model of Vulić and Moens (2016) which learns word-level bilingual embeddings from document-aligned comparable data (BWESG). For both models, the top layers of our proposed classification network should learn to relate the word-level features stemming from these word embeddings using a set of annotated translation pairs.…”

“…Therefore, we have resorted to a heuristic which uses a noisy classifier: it generates 2N c << |V T | translation candidates for each source language word w S as follows. It generates (1) the N c target words closest to w S measured by edit distance as translations, and (2) N c target words measured closest to w S based on the cosine distance between their wordlevel embeddings in a bilingual space induced by the embedding model of Vulić and Moens (2016). …”

“…In this work, we experiment with two BWE models that have demonstrated a strong BLI performance using only a small seed set of word translation pairs (Mikolov et al, 2013b), or document alignments (Vulić and Moens, 2016) for bilingual supervision.…”

Learning bilingual word embeddings with (almost) no bilingual data

“…A possible relaxation is to use document-aligned or label-aligned comparable corpora (Søgaard et al, 2015;Vulić and Moens, 2016;Mogadala and Rettinger, 2016), but large amounts of such corpora are not always available for some language pairs.…”