NORMA: Neighborhood Sensitive Maps for Multilingual Word Embeddings

Abstract

Inducing multilingual word embeddings by learning a linear map between embedding spaces of different languages achieves remarkable accuracy on related languages. However, accuracy drops substantially when translating between distant languages. Given that languages exhibit differences in vocabulary, grammar, written form, or syntax, one would expect that embedding spaces of different languages have different structures especially for distant languages. With the goal of capturing such differences, we propose a method for learning neighborhood sensitive maps, NORMA. Our experiments show that NORMA outperforms current state-of-the-art methods for word translation between distant languages.

A Margin-based Loss with Synthetic Negative Samples for Continuous-output Machine Translation

“…We build on (Kumar and Tsvetkov, 2019), one of the first instances of using pretrained embeddings as model outputs for complex sequencegeneration tasks. Closely related work on embedding prediction includes zero-shot learning for word translation (Nakashole, 2018;Conneau et al, 2018) and image labeling (Lazaridou et al, 2015), as well as rare word prediction (Pinter et al, 2018) and classification (Card et al, 2019).…”

How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions

“…In addition, recent work has leveraged topological similarities between monolingual vector spaces to introduce fully unsupervised projectionbased CLE approaches that do not require any bilingual supervision (Zhang et al, 2017;Conneau et al, 2018a;Artetxe et al, 2018b;Alvarez-Melis and Jaakkola, 2018). Being conceptually attractive, such weakly supervised and unsupervised CLEs have taken the field by storm recently (Conneau et al, 2018a;Dou et al, 2018;Doval et al, 2018;Hoshen and Wolf, 2018;Kim et al, 2018;Chen and Cardie, 2018;Mukherjee et al, 2018;Nakashole, 2018;Xu et al, 2018;Alaux et al, 2019).…”

Supervised and Nonlinear Alignment of Two Embedding Spaces for Dictionary Induction in Low Resourced Languages

“…However, the main success of these methods was obtained on linguistically similar languages where a large volume of monolingual data from compatible sources is available. A recent study showed that a simple linear mapping is not enough for low resourced languages (Nakashole, 2018). However, the proposed method in this study, NorMA, came short in solving the problem showing significant gains in mapping precision for Russian-English and Chinese-English, compared not to the best model in (Lample et al, 2018), while underperforming in Spanish-English French-English and German-English.…”

“…In low resourced languages with a significantly lower amount of data compared to the English corpora structural similarities of the two spaces are not guaranteed leading to weak initial assumptions. This is also true in languages with lower similarity to English such as Slavic and Asian languages (Nakashole, 2018;Nakashole and Flauger, 2018). Nakashole and Flauger showed that word neighborhoods require different linear mappings to correctly capture the word relation.…”

“…Initial success obtained in aligning two embedding spaces has created a flurry of research efforts into improving the task (Nakashole, 2018;Doval et al, 2018;Kementchedjhieva et al, 2018;Chen and Cardie, 2018;Hoshen and Wolf, 2018;Artetxe et al, 2018), to name a few published in 2018. There are two main approaches for mapping a monolingual word embedding space into another (Ruder et al, 2017): 1) Supervised methods rely on a few anchor points to learn the matching of two spaces (Lample et al, 2018;Mikolov et al, 2013;Xing et al, 2015; and; 2) Unsupervised methods mostly formulate the problem as an adversarial optimization problem (Lample et al, 2018;Zhang et al, 2017;Ruder et al, 2017;Chen and Cardie, 2018;Hoshen and Wolf, 2018).…”