Are Girls Neko or Shōjo? Cross-Lingual Alignment of Non-Isomorphic Embeddings with Iterative Normalization

Abstract

Cross-lingual word embeddings (CLWE) underlie many multilingual natural language processing systems, often through orthogonal transformations of pre-trained monolingual embeddings. However, orthogonal mapping only works on language pairs whose embeddings are naturally isomorphic. For nonisomorphic pairs, our method (Iterative Normalization) transforms monolingual embeddings to make orthogonal alignment easier by simultaneously enforcing that (1) individual word vectors are unit length, and (2) each language's average vector is zero. Iterative Normalization consistently improves word translation accuracy of three CLWE methods, with the largest improvement observed on English-Japanese (from 2% to 44% test accuracy).

Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?

“…Unsupervised approaches even exploit the assumption twice as their seed extraction is fully based on the topological similarity. Future work should move beyond the restrictive assumption by exploring new methods that can, e.g., 1) increase the isomorphism between monolingual spaces (Zhang et al, 2019) by distinguishing between language-specific and language-pairinvariant subspaces; 2) learn effective non-linear or multiple local projections between monolingual spaces similar to the preliminary work of Nakashole (2018); 3) similar to Vulić and Korhonen (2016) and Lubin et al ( 2019) "denoisify" seed lexicons during the self-learning procedure. For instance, keeping only mutual/symmetric nearest neighbour as in FULL+SL+SYM can be seen as a form of rudimentary denoisifying: it is indicative to see that the best overall performance in this work is reported with that model configuration.…”

Are All Good Word Vector Spaces Isomorphic?

“…Relational similarity As an alternative, we consider a simpler measure inspired by Zhang et al (2019). This measure, dubbed RSIM, is based on the intuition that the similarity distributions of translations within each language should be similar.…”

“…Corpus size It has become standard to align monolingual word embeddings trained on Wikipedia Zhang et al, 2019). As can be seen in Figure 1, and also in Table 1, Wikipedias of low-resource languages are more than a magnitude smaller than Wikipedias of high-resource languages.…”

“…Preprocessing Different forms of preprocessing have been shown to aid in learning a mapping (Artetxe et al, 2018b;Zhang et al, 2019). Consequently, they may also influence the isomorphism of the vector spaces.…”

“…In fact, the amount of data used to induce the monolingual embeddings is predictive of the quality of the aligned cross-lingual word embeddings, as evaluated on bilingual lexicon induction (BLI). Consider, for motivation, Figure 1; it shows the performance of a state-of-the-art alignment method-RCSLS with iterative normalisation (Zhang et al, 2019)-on mapping English embeddings onto embeddings in other languages, and its correlation (ρ = 0.72) with the size of the tokenised target language Polyglot Wikipedia (Al-Rfou et al, 2013).…”