On Dimensional Linguistic Properties of the Word Embedding Space

Abstract

Word embeddings have become a staple of several natural language processing tasks, yet much remains to be understood about their properties. In this work, we analyze word embeddings in terms of their principal components and arrive at a number of novel and counterintuitive observations. In particular, we characterize the utility of variance explained by the principal components as a proxy for downstream performance. Furthermore, through syntactic probing of the principal embedding space, we show that the syntactic information captured by a principal component does not correlate with the amount of variance it explains. Consequently, we investigate the limitations of variance based embedding post-processing, used in a few algorithms such as (Mu and Viswanath, 2018;Raunak et al., 2019) and demonstrate that such postprocessing is counter-productive in sentence classification and machine translation tasks. Finally, we offer a few precautionary guidelines on applying variance based embedding post-processing and explain why non-isotropic geometry might be integral to word embedding performance.

Autoencoding Improves Pre-trained Word Embeddings

“…However, the frequency of a word is an important piece of information for tasks that require differentiating stop words and content words such as in information retrieval. Raunak et al (2020) demonstrated that removing the top principal components does not necessarily lead to performance improvement. Moreover, contextualised word embeddings such as BERT (Devlin et al, 2019) and Elmo (Peters et al, 2018) have shown to be anisotropic despite their superior performance in a wide-range of NLP tasks (Ethayarajh, 2019).…”

Exploring Dimensionality Reduction Techniques in Multilingual Transformers

“…Alternatively, using self-supervised trained checkpoints of these models and their pre-trained embeddings as a starting point to be later fine-tuned for supervised downstream tasks is widely used. Unlike previous works in the literature that have only focused on reduced pre-trained embeddings [25,26,27], in this work we are interested of the evaluation of the impact of dimensionality reduction on both types of embeddings, pre-trained and downstream fine-tuned embeddings.…”

“…Recently, the study of Raunak et al [25,26] has shed more light on the importance of reducing the size of embeddings produced by Machine Learning and Deep Learning models. More specifically, these authors draw attention to reducing the size of classical GloVe [32] and FastText [33] pre-trained word embeddings using PCA-based post-processing algorithms, achieving similar or even better performance than the original embeddings.…”

“…Hence, the presented work follows the research line proposed by different authors [25,26,27,39] but takes a step forward, including a broader range of techniques and evaluating the capability of dimensionality reduction techniques in both pre-trained and fine-tuned pre-computed embeddings from state-of-the-art contextual-based Transformer models from the recently claimed multilingual point of view.…”

Autoencoding Improves Pre-trained Word Embeddings

“…However, the frequency of a word is an important piece of information for tasks that require differentiating stop words and content words such as in information retrieval. Actually, Raunak et al (2020) demonstrated that removing the top principal components does not necessarily lead to performance improvement. Moreover, contextualised word embeddings such as BERT (Devlin et al, 2019) and Elmo (Peters et al, 2018) have shown to be anisotropic despite their superior performance in a wide-range of NLP tasks (Ethayarajh, 2019).…”