Unsupervised Multilingual Word Embedding with Limited Resources using Neural Language Models

Abstract

We propose an unsupervised method to obtain cross-lingual embeddings without any parallel data or pre-trained word embeddings. The proposed model, which we call multilingual neural language models, takes sentences of multiple languages as an input. The proposed model contains bidirectional LSTMs that perform as forward and backward language models, and these networks are shared among all the languages. The other parameters, i.e. word embeddings and linear transformation between hidden states and outputs, are specific to each language. The shared LSTMs can capture the common sentence structure among all languages. Accordingly, word embeddings of each language are mapped into a common latent space, making it possible to measure the similarity of words across multiple languages. We evaluate the quality of the cross-lingual word embeddings on a word alignment task. Our experiments demonstrate that our model can obtain cross-lingual embeddings of much higher quality than existing unsupervised models when only a small amount of monolingual data (i.e. 50k sentences) are available, or the domains of monolingual data are different across languages.

Multilingual Denoising Pre-training for Neural Machine Translation

“…We bold the highest transferring score for each pair. Multilinguality in NLP tasks This work is also related to the continual trend of multilingual language learning, including aligning multilingual word embeddings (Mikolov et al, 2013;Chen and Cardie, 2018;Lample et al, 2018b) into universal space, and learning crosslingual models (Wada and Iwata, 2018;Lample and Conneau, 2019; to exploit shared representations across languages.…”

Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary Induction

“…Subsets of the MUSE dictionaries have been used for model comparison in the evaluation of numerous cross-lingual embedding systems developed since (cf. Jawanpuria et al, 2018;Hoshen and Wolf, 2018a,b;Wada and Iwata, 2018;. Even though the field has been very active, progress has been incremental for most language pairs.…”

Hierarchical Mapping for Crosslingual Word Embedding Alignment

“…Attempts to do so include the efforts by , who leverage an inverted index based on the Wikipedia multilingual links to generate multilingual word representations. Wada et al (2019) instead use a sentence-level neural language model for directly learning multilingual word embeddings and as a result bypassing the need for mapping functions. In the paradigm of aligning pre-trained word embeddings where we focus, Heyman et al (2019) propose a technique that iteratively builds a multilingual space starting from a monolingual space and incrementally incorporating languages to it.…”