Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?

Abstract

Recent efforts in cross-lingual word embedding (CLWE) learning have predominantly focused on fully unsupervised approaches that project monolingual embeddings into a shared cross-lingual space without any cross-lingual signal. The lack of any supervision makes such approaches conceptually attractive. Yet, their only core difference from (weakly) supervised projection-based CLWE methods is in the way they obtain a seed dictionary used to initialize an iterative self-learning procedure. The fully unsupervised methods have arguably become more robust, and their primary use case is CLWE induction for pairs of resourcepoor and distant languages. In this paper, we question the ability of even the most robust unsupervised CLWE approaches to induce meaningful CLWEs in these more challenging settings. A series of bilingual lexicon induction (BLI) experiments with 15 diverse languages (210 language pairs) show that fully unsupervised CLWE methods still fail for a large number of language pairs (e.g., they yield zero BLI performance for 87/210 pairs). Even when they succeed, they never surpass the performance of weakly supervised methods (seeded with 500-1,000 translation pairs) using the same self-learning procedure in any BLI setup, and the gaps are often substantial. These findings call for revisiting the main motivations behind fully unsupervised CLWE methods.

Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining

“…Several recent studies (Patra et al, 2019;Ormazabal et al, 2019) criticize this simplified approach, showing that even the embedding spaces of closely related languages are not isometric. Vulić et al (2019) question the robustness of unsupervised mapping methods in challenging circumstances.…”

LLOD-Driven Bilingual Word Embeddings Rivaling Cross-Lingual Transformers in Quality of Life Concept Detection from French Online Health Communities

“…As our research questions imply the availability of applicable lexica, unsupervised or weakly supervised approaches for inducing bilingual word embeddings [9,10,11] are only indirectly relevant to our work. However, we plan to compare against them in future work, especially given that claims of comparable or even superior performance of unsupervised methods (e.g., [12]) have been called into question [13,14], in particular when evaluated on actual downstream tasks instead of bilingual lexicon induction [15].…”

Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation

“…While massively multilingual models have obtained impressive quality improvements for low-resource languages as well as zero-shot scenarios (Aharoni et al, 2019;Arivazhagan et al, 2019a), it has not yet been shown how these massively multilingual models could be extended to unseen languages, beyond the pipelined approaches (Currey and Heafield, 2019;Lakew et al, 2019). On the other hand, self-supervised learning approaches have excelled at down-stream cross-lingual transfer (Devlin et al, 2019;Raffel et al, 2019;, but their success for unsupervised NMT (Conneau and Lample, 2019;Song et al, 2019) currently lacks robustness when languages are distant or monolingual data domains are mismatched (Neubig and Hu, 2018;Vulić et al, 2019). We observe that these two lines of research can be quite complementary and can compensate for each other's deficiencies.…”