Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction

Zhang, Jinpeng; Ji, Baijun; Xiao, Nini; Duan, Xiangyu; Zhang, Min; Shi, Yangbin; Luo, Weihua

doi:10.18653/v1/2021.findings-acl.260

Cited by 10 publications

(7 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CLWE approaches involve some levels of supervised alignment (Faruqui and Dyer, 2014;Zou et al, 2013), seed dictionaries (Artetxe et al, 2017;Gouws and Søgaard, 2015) or adversarial training (Lample et al, 2018;Artetxe et al, 2018;Zhang et al, 2017;Miceli Barone, 2016). Contexualized embeddings from language models have also been used in combination with static word embeddings to improve alignments of crosslingual word vectors (Aldarmaki and Diab, 2019;Zhang et al, 2021). Contrary to our findings for the token embeddings of LLMs, it was not clear that aligning word vectors is possible without some level of supervision, or to more than two languages at a time.…”

Section: Related Workcontrasting

confidence: 75%

Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings

Wen-Yi,

Mimno

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Cross-lingual transfer learning is an important property of multilingual large language models (LLMs). But how do LLMs represent relationships between languages? Every language model has an input layer that maps tokens to vectors. This ubiquitous layer of language models is often overlooked. We find that similarities between these input embeddings are highly interpretable and that the geometry of these embeddings differs between model families. In one case (XLM-RoBERTa), embeddings encode language: tokens in different writing systems can be linearly separated with an average of 99.2% accuracy. Another family (mT5) represents cross-lingual semantic similarity: the 50 nearest neighbors for any token represent an average of 7.61 writing systems, and are frequently translations. This result is surprising given that there is no explicit parallel crosslingual training corpora and no explicit incentive for translations in pre-training objectives. Our research opens the door for investigations in 1) The effect of pre-training and model architectures on representations of languages and 2) The applications of cross-lingual representations embedded in language models.

show abstract

Section: Related Workcontrasting

confidence: 75%

Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings

Wen-Yi,

Mimno

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…Most commonly, due to reduced bilingual supervision requirements, the CLWEs are induced by (i) pretraining monolingual word embeddings independently in two languages, and then (ii) mapping them by linear (Mikolov et al, 2013;Xing et al, 2015;Joulin et al, 2018;Artetxe et al, 2018) or non-linear transformations Mohiuddin et al, 2020), minimising the distance between the original monolingual word embedding spaces. Optionally, such static CLWEs can be combined or enhanced with external word-level knowledge such as word translation knowledge embedded in multilingual language models (Zhang et al, 2021;Li et al, 2022;.…”

Section: Methodsmentioning

confidence: 99%

“…However, empirical evidence suggests that these approaches underperform static CLWEs for BLI (Vulić et al, 2020b): this is possibly because PLMs are primarily designed for longer sequence-level tasks and thus may naturally have inferior performance in word-level tasks when used off-the-shelf . Recent work started to combine static and contextualised word representations for BLI (Zhang et al, 2021). In fact, the previous SotA CLWEs for BLI, used as the baseline model in our work, are derived via a two-stage contrastive learning approach combining word representations of both types (Li et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

Improving Bilingual Lexicon Induction with Cross-Encoder Reranking

Li¹,

Liu²,

Vulić³

et al. 2022

Preprint

View full text Add to dashboard Cite

Bilingual lexicon induction (BLI) with limited bilingual supervision is a crucial yet challenging task in multilingual NLP. Current state-ofthe-art BLI methods rely on the induction of cross-lingual word embeddings (CLWEs) to capture cross-lingual word similarities; such CLWEs are obtained 1) via traditional static models (e.g., VECMAP), or 2) by extracting type-level CLWEs from multilingual pretrained language models (mPLMs), or 3) through combining the former two options. In this work, we propose a novel semi-supervised post-hoc reranking method termed BLICEr (BLI with Cross-Encoder Reranking), applicable to any precalculated CLWE space, which improves their BLI capability. The key idea is to 'extract' cross-lingual lexical knowledge from mPLMs, and then combine it with the original CLWEs. This crucial step is done via 1) creating a word similarity dataset, comprising positive word pairs (i.e., true translations) and hard negative pairs induced from the original CLWE space, and then 2) fine-tuning an mPLM (e.g., mBERT or XLM-R) in a crossencoder manner to predict the similarity scores. At inference, we 3) combine the similarity score from the original CLWE space with the score from the BLI-tuned cross-encoder. BLICEr establishes new state-of-the-art results on two standard BLI benchmarks spanning a wide spectrum of diverse languages: it substantially outperforms a series of strong baselines across the board. We also validate the robustness of BLICEr with different CLWEs.

show abstract

“…When dealing with systems that have the potential to leverage dynamic word representations from various language models (e.g., CLBLI), we do not integrate these dynamic word representations. Because the advantages of static WEs and dynamic word representations bring in enhancing BLI are orthogonal (Zhang et al 2021).…”

Section: Training Setup and Hyperparametersmentioning

confidence: 99%

Enhancing Bilingual Lexicon Induction via Bi-directional Translation Pair Retrieving

Ding,

Cao,

Zhao

2024

AAAI

View full text Add to dashboard Cite

Most Bilingual Lexicon Induction (BLI) methods retrieve word translation pairs by finding the closest target word for a given source word based on cross-lingual word embeddings (WEs). However, we find that solely retrieving translation from the source-to-target perspective leads to some false positive translation pairs, which significantly harm the precision of BLI. To address this problem, we propose a novel and effective method to improve translation pair retrieval in cross-lingual WEs. Specifically, we consider both source-side and target-side perspectives throughout the retrieval process to alleviate false positive word pairings that emanate from a single perspective. On a benchmark dataset of BLI, our proposed method achieves competitive performance compared to existing state-of-the-art (SOTA) methods. It demonstrates effectiveness and robustness across six experimental languages, including similar language pairs and distant language pairs, under both supervised and unsupervised settings.

show abstract

Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction

Cited by 10 publications

References 28 publications

Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings

Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings

Improving Bilingual Lexicon Induction with Cross-Encoder Reranking

Enhancing Bilingual Lexicon Induction via Bi-directional Translation Pair Retrieving

Contact Info

Product

Resources

About