How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks

Jastrzebski, Stanisław; Leśniak, Damian; Czarnecki, Wojciech Marian

doi:10.48550/arxiv.1702.02170

Cited by 16 publications

(19 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…GloVe is trained for 100 epochs and parameters are set to those recommended in the paper (x-max is 100 and context window size is 10). We used open-source word embedding evaluation tool kits for assessing the quality of the models (Jastrzebski et al, 2017;Conneau and Kiela, 2018). Table 4: Interpretation of attention.…”

Section: Methodsmentioning

confidence: 99%

Attention Word Embedding

Sonkar¹,

Waters²,

Baraniuk³

2020

Preprint

View full text Add to dashboard Cite

Word embedding models learn semantically rich vector representations of words and are widely used to initialize natural processing language (NLP) models. The popular continuous bag-ofwords (CBOW) model of word2vec learns a vector embedding by masking a given word in a sentence and then using the other words as a context to predict it. A limitation of CBOW is that it equally weights the context words when making a prediction, which is inefficient, since some words have higher predictive value than others. We tackle this inefficiency by introducing the Attention Word Embedding (AWE) model, which integrates the attention mechanism into the CBOW model. We also propose AWE-S, which incorporates subword information. We demonstrate that AWE and AWE-S outperform the state-of-the-art word embedding models both on a variety of word similarity datasets and when used for initialization of NLP models.

show abstract

Section: Methodsmentioning

confidence: 99%

Attention Word Embedding

Sonkar¹,

Waters²,

Baraniuk³

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The word pairs are selected from the following benchmark datasets: RG (Rubenstein and Goodenough, 1965), MTurk (Radinsky et al, 2011), RW (Luong et al, 2013), MEN (Bruni et al, 2014), SimLex999 (Hill et al, 2015) and AP (Almuhareb and Poesio, 2005). The results for these tests are obtained from the word embedding benchmark package (Jastrzebski et al, 2017) 7 . Note that it is not our primary aim to achieve a state-of-the-art result in this test.…”

Section: Word Semantic Similarity Testmentioning

confidence: 99%

Nurse is Closer to Woman than Surgeon? Mitigating Gender-Biased Proximities in Word Embeddings

Kumar

Bhotia

Chakraborty

2020

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

Word embeddings are the standard model for semantic and syntactic representations of words. Unfortunately, these models have been shown to exhibit undesirable word associations resulting from gender, racial, and religious biases. Existing post-processing methods for debiasing word embeddings are unable to mitigate gender bias hidden in the spatial arrangement of word vectors. In this paper, we propose RAN-Debias, a novel gender debiasing methodology that not only eliminates the bias present in a word vector but also alters the spatial distribution of its neighboring vectors, achieving a bias-free setting while maintaining minimal semantic offset. We also propose a new bias evaluation metric, Gender-based Illicit Proximity Estimate (GIPE), which measures the extent of undue proximity in word vectors resulting from the presence of gender-based predilections. Experiments based on a suite of evaluation metrics show that RAN-Debias significantly outperforms the state-of-the-art in reducing proximity bias (GIPE) by at least 42.02%. It also reduces direct bias, adding minimal semantic disturbance, and achieves the best performance in a downstream application task (coreference resolution).

show abstract

“…Many benchmarks have been proposed for the evaluation of unsupervised word representations. In general, they can be divided into intrinsic and extrinsic evaluation methods (Schnabel et al, 2015;Chiu et al, 2016;Jastrzebski et al, 2017;Alshargi et al, 2018;Bakarov, 2018). While most datasets report the semantic similarity between words, many datasets actually capture semantic relatedness (Hill et al, 2015;Avraham and Goldberg, 2016), or more complex relations such as analogy or the ability to categorize words based on the distributed representation encoded in word embeddings.…”

Section: Word Similarity and Relatednessmentioning

confidence: 99%

Syntactic Interchangeability in Word Embedding Models

Hershcovich

Toledo

Halfon

et al. 2019

Proceedings of the 3rd Workshop on Evaluating Vector Space Representations For

View full text Add to dashboard Cite

Nearest neighbors in word embedding models are commonly observed to be semantically similar, but the relations between them can vary greatly. We investigate the extent to which word embedding models preserve syntactic interchangeability, as reflected by distances between word vectors, and the effect of hyper-parameters-context window size in particular. We use part of speech (POS) as a proxy for syntactic interchangeability, as generally speaking, words with the same POS are syntactically valid in the same contexts. We also investigate the relationship between interchangeability and similarity as judged by commonly-used word similarity benchmarks, and correlate the result with the performance of word embedding models on these benchmarks. Our results will inform future research and applications in the selection of word embedding model, suggesting a principle for an appropriate selection of the context window size parameter depending on the use-case.

show abstract

How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks

Cited by 16 publications

References 0 publications

Attention Word Embedding

Attention Word Embedding

Nurse is Closer to Woman than Surgeon? Mitigating Gender-Biased Proximities in Word Embeddings

Syntactic Interchangeability in Word Embedding Models

Contact Info

Product

Resources

About