Are All Good Word Vector Spaces Isomorphic?

Vulić, Ivan; Ruder, Sebastian; Søgaard, Anders

doi:10.18653/v1/2020.emnlp-main.257

Cited by 29 publications

(26 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The differences in embedding spaces of different languages do not only depend on linguistic properties of the languages in consideration, but also on other factors such as the chosen training algorithm, underlying training domain, or training data size and quality Arora et al, 2019;Vulić et al, 2020). In future research we also plan an in-depth study of these factors and their relation to our spectral analysis.…”

Section: Further Discussion and Conclusionmentioning

confidence: 97%

The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures

Dubossarsky¹,

Vulić²,

Reichart³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

Performance in cross-lingual NLP tasks is impacted by the (dis)similarity of languages at hand: e.g., previous work has suggested there is a connection between the expected success of bilingual lexicon induction (BLI) and the assumption of (approximate) isomorphism between monolingual embedding spaces. In this work we present a large-scale study focused on the correlations between monolingual embedding space similarity and task performance, covering thousands of language pairs and four different tasks: BLI, parsing, POS tagging and MT. We hypothesize that statistics of the spectrum of each monolingual embedding space indicate how well they can be aligned. We then introduce several isomorphism measures between two embedding spaces, based on the relevant statistics of their individual spectra. We empirically show that 1) language similarity scores derived from such spectral isomorphism measures are strongly associated with performance observed in different crosslingual tasks, and 2) our spectral-based measures consistently outperform previous standard isomorphism measures, while being computationally more tractable and easier to interpret. Finally, our measures capture complementary information to typologically driven language distance measures, and the combination of measures from the two families yields even higher task performance correlations.

show abstract

Section: Further Discussion and Conclusionmentioning

confidence: 97%

The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures

Dubossarsky¹,

Vulić²,

Reichart³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Our method can be adjusted for multitask and multilingual settings. Following the observation that the orthogonal transformation can map distributions of embeddings in typologically close languages (Mikolov et al, 2013;Vulić et al, 2020). We think that joint training for many languages may be possible by keeping the same Scaling Vector and adding a separate Orthogonal Transformation per language, fulfilling the role of orthogonal mappings.…”

Section: Further Workmentioning

confidence: 93%

Introducing Orthogonal Constraint in Structural Probes

Limisiewicz¹,

Mareek²

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

With the recent success of pre-trained models in NLP, a significant focus was put on interpreting their representations. One of the most prominent approaches is structural probing (Hewitt and Manning, 2019), where a linear projection of word embeddings is performed in order to approximate the topology of dependency structures. In this work, we introduce a new type of structural probing, where the linear projection is decomposed into 1. isomorphic space rotation; 2. linear scaling that identifies and scales the most relevant dimensions. In addition to syntactic dependency, we evaluate our method on novel tasks (lexical hypernymy and position in a sentence). We jointly train the probes for multiple tasks and experimentally show that lexical and syntactic information is separated in the representations. Moreover, the orthogonal constraint makes the Structural Probes less vulnerable to memorization.

show abstract

“…An empirical measure of semantic proximity between two languages is often computed using the degree of isomorphism, that is, how similar the structures of two languages are in topological space (Søgaard et al, 2018). Research in cross-lingual transfer tasks shows that linguistic differences across languages often make spaces depart from isomorphism (Nakashole and Flauger, 2018;Søgaard et al, 2018;Patra et al, 2019;Vulić et al, 2020). While this degrades the quality of bilingual embeddings, it is a desired characteristic in our case: since our task involves processing of (multi-view) representations of monolingual text, departures from isomorphism indicate diversity in the source that generates them.…”

Section: Measuring Isomorphismmentioning

confidence: 99%

Understanding Translationese in Multi-view Embedding Spaces

Chowdhury¹,

España-Bonet

Genabith

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Recent studies use a combination of lexical and syntactic features to show that footprints of the source language remain visible in translations, to the extent that it is possible to predict the original source language from the translation. In this paper, we focus on embedding-based semantic spaces, exploiting departures from isomorphism between spaces built from original target language and translations into this target language to predict relations between languages in an unsupervised way. We use different views of the data -words, parts of speech, semantic tags and synsets -to track translationese. Our analysis shows that (i) semantic distances between original target language and translations into this target language can be detected using the notion of isomorphism, (ii) language family ties with characteristics similar to linguistically motivated phylogenetic trees can be inferred from the distances and (iii) with delexicalised embeddings exhibiting source-language interference most significantly, other levels of abstraction display the same tendency, indicating the lexicalised results to be not "just" due to possible topic differences between original and translated texts. To the best of our knowledge, this is the first time departures from isomorphism between embedding spaces are used to track translationese.

show abstract

Are All Good Word Vector Spaces Isomorphic?

Cited by 29 publications

References 36 publications

The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures

The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures

Introducing Orthogonal Constraint in Structural Probes

Understanding Translationese in Multi-view Embedding Spaces

Contact Info

Product

Resources

About