On the Role of Seed Lexicons in Learning Bilingual Word Embeddings

Abstract

A shared bilingual word embedding space (SBWES) is an indispensable resource in a variety of cross-language NLP and IR tasks. A common approach to the SB-WES induction is to learn a mapping function between monolingual semantic spaces, where the mapping critically relies on a seed word lexicon used in the learning process. In this work, we analyze the importance and properties of seed lexicons for the SBWES induction across different dimensions (i.e., lexicon source, lexicon size, translation method, translation pair reliability). On the basis of our analysis, we propose a simple but effective hybrid bilingual word embedding (BWE) model. This model (HYBWE) learns the mapping between two monolingual embedding spaces using only highly reliable symmetric translation pairs from a seed document-level embedding space. We perform bilingual lexicon learning (BLL) with 3 language pairs and show that by carefully selecting reliable translation pairs our new HYBWE model outperforms benchmarking BWE learning models, all of which use more expensive bilingual signals. Effectively, we demonstrate that a SBWES may be induced by leveraging only a very weak bilingual signal (document alignments) along with monolingual data.

Knowledge Distillation for Bilingual Dictionary Induction

“…• The method Ridge is used in a number of prior work (Mikolov et al, 2013;Dinu et al, 2014;Vulic and Korhonen, 2016). These approaches use an L2-regularized least-squares error objective as shown in Equation 2.…”

“…Our work is most related to models that do zero-shot learning for bilingual dictionary induction, using maps between vector spaces with seed dictionaries as training data. Examples include the models of (Mikolov et al, 2013;Dinu et al, 2014;Lazaridou et al, 2015;Vulic and Korhonen, 2016). Like these approaches, we first learn word embeddings for each language, then use a seed dictionary to train a mapping function between the two vector spaces.…”

“…Additionally, we model linguistic information in the vector space of the source and target languages. Another line of research in this vein is the work of (Vulic and Korhonen, 2016), who analyze how properties of the seed dictionary affect bilingual dictionary induction across different dimensions (i.e., lexicon source, lexicon size, translation method, translation pair reliability). However, methodologically, their approach is based on prior work (Mikolov et al, 2013;Dinu et al, 2014).…”

“…Bilingual word embedding learning methods produce a shared bilingual word embedding space where words from two languages are represented in the new space so that similar words, which may be in different languages, have similar representations. Such bilingual word embeddings have been used in a number of tasks including semantic word similarity (Faruqui and Dyer, 2014;Ammar et al, 2016) learning bilingual word lexicons (Mikolov et al, 2013;Gouws et al, 2015;Vulic and Korhonen, 2016), parsing (Guo et al, 2015;Täckström et al, 2012), information retrieval (Vulic and Moens, 2015), and cross-lingual document classification (Klementiev et al, 2012;Kočiskỳ et al, 2014). Some bilingual word embedding methods such as Gouws et al, 2015) require sentence or word aligned data, which our approach does not require.…”

Learning Word Subsumption Projections for the Russian Language

“…More recently, Vulic et al [8] presented a systematic study of four classes of methods for learning bilingual embeddings. The authors find approach based on linear projection, similar to the one we use in our method, to be most practical and efficient.…”