Obtaining Better Static Word Embeddings Using Contextual Embedding Models

Abstract

The advent of contextual word embeddingsrepresentations of words which incorporate semantic and syntactic information from their context-has led to tremendous improvements on a wide variety of NLP tasks. However, recent contextual models have prohibitively high computational cost in many use-cases and are often hard to interpret. In this work, we demonstrate that our proposed distillation method, which is a simple extension of CBOW-based training, allows to significantly improve computational efficiency of NLP applications, while outperforming the quality of existing static embeddings trained from scratch as well as those distilled from previously proposed methods. As a side-effect, our approach also allows a fair comparison of both contextual and static embeddings via standard lexical evaluation tasks.

Combining Static and Contextualised Multilingual Embeddings

“…This difference allows us to apply the alignment to many more languages than most related work. For example, Wang et al (2019) 3 Static Embeddings from XLM-R Gupta and Jaggi (2021) extracted English static embeddings from BERT and RoBERTa. They showed that their CBOW-like training scales better with more data and outperforms an aggregation approach to extracting static embeddings (Bommasani et al, 2020).…”

“…We choose 40 languages for static embeddings extraction (full list in Appendix A). As the multilingual contextual model, we use XLM-R. From preliminary experimentation, we determined how best to extract multilingual embeddings from the model: First, using X2Static (Gupta and Jaggi, 2021) worked better than aggregation (Bommasani et al, 2020) even with a small amount of data. One important difference with Gupta and Jaggi's work is that for our task the sentence-level variant of X2Static worked better than the paragraph-level version.…”

“…Monolingual static embeddings have been extracted from BERT by Gupta and Jaggi (2021). We show that their approach can be applied to multilingual embeddings.…”

An Empirical Study on the Fairness of Pre-trained Word Embeddings

“…One of the goals of the paper selection was to extract the most relevant pre-trained word embedding models from the many that have been studied. While recent research on contextual embeddings has proven immensely beneficial, static embeddings remain crucial in many situations (Gupta and Jaggi, 2021). Many NLP applications fundamentally depend on static word embeddings for metrics that are designed non-contextual (Shoemark et al, 2019), such as examining word vector spaces (Vulic et al, 2020) and bias study (Gonen and Goldberg, 2019;Kaneko and Bollegala, 2019;Manzini et al, 2019).…”

Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)

“…One of the goals of the paper selection was to extract the most relevant pre-trained word embedding models from the many that have been studied. While recent research on contextual embeddings has proven immensely beneficial, static embeddings remain crucial in many situations (Gupta and Jaggi, 2021). Many NLP applications fundamentally depend on static word embeddings for metrics that are designed non-contextual (Shoemark et al, 2019), such as examining word vector spaces (Vulic et al, 2020) and bias study Manzini et al, 2019).…”