Placing search in context

Finkelstein, Lev; Gabrilovich, Evgeniy; Matias, Yossi; Rivlin, Ehud; Solan, Zach; Wolfman, Gadi; Ruppin, Eytan

doi:10.1145/371920.372094

Cited by 604 publications

(80 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First we consider a group of word-level similarity datasets that are commonly used as benchmarks in previous research: WS-353-SIM (Finkelstein et al, 2001), YP-130 (Yang and Powers, 2005), SIMLEX-999 (Hill et al, 2015), SimVerb-3500 (Gerz et al, 2016), RW-STANFORD (Luong Table 1: Spearman's ρ on word similarity tasks for combinations of word vectors and the following similarity metrics: cosine similarity (COS), Pearson's r (PRS), Spearman's ρ (SPR), and Kendall τ (KEN). N indicates the proportion of sentence vectors in a task for which the null hypothesis of normality in a Shapiro-Wilk test was not rejected at α = 0.05.…”

Section: Methodsmentioning

confidence: 99%

Correlation Coefficients and Semantic Textual Similarity

Железняк¹,

Savkov

Shen³

et al. 2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

A large body of research into semantic textual similarity has focused on constructing state-of-the-art embeddings using sophisticated modelling, careful choice of learning signals and many clever tricks. By contrast, little attention has been devoted to similarity measures between these embeddings, with cosine similarity being used unquestionably in the majority of cases. In this work, we illustrate that for all common word vectors, cosine similarity is essentially equivalent to the Pearson correlation coefficient, which provides some justification for its use. We thoroughly characterise cases where Pearson correlation (and thus cosine similarity) is unfit as similarity measure. Importantly, we show that Pearson correlation is appropriate for some word vectors but not others. When it is not appropriate, we illustrate how common nonparametric rank correlation coefficients can be used instead to significantly improve performance. We support our analysis with a series of evaluations on word-level and sentencelevel semantic textual similarity benchmarks. On the latter, we show that even the simplest averaged word vectors compared by rank correlation easily rival the strongest deep representations compared by cosine similarity.

show abstract

Section: Methodsmentioning

confidence: 99%

Correlation Coefficients and Semantic Textual Similarity

Железняк¹,

Savkov

Shen³

et al. 2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

show abstract

“…The correlation between the human ratings and similarity scores computed using word embeddings for pairs of words has been used as a measure of the quality of the word embeddings (Mikolov et al, 2013d). We compute cosine similarity between word embeddings and measure Spearman correlation against human rat-ings for the word-pairs in the following benchmark datasets: Word Similarity 353 dataset (WS) (Finkelstein et al, 2001), Rubenstein-Goodenough dataset (RG) (Rubenstein and Goodenough, 1965), MTurk (Halawi et al, 2012), rare words dataset (RW) (Luong et al, 2013), MEN dataset (Bruni et al, 2012) and SimLex dataset (Hill et al, 2015). Unfortunately, existing benchmark datasets for semantic similarity were not created considering gender-biases and contain many stereotypical examples.…”

Section: Semantic Similarity Measurementmentioning

confidence: 99%

Gender-preserving Debiasing for Pre-trained Word Embeddings

Kaneko¹,

Bollegala²

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Word embeddings learnt from massive text collections have demonstrated significant levels of discriminative biases such as gender, racial or ethnic biases, which in turn bias the down-stream NLP applications that use those word embeddings. Taking gender-bias as a working example, we propose a debiasing method that preserves non-discriminative gender-related information, while removing stereotypical discriminative gender biases from pre-trained word embeddings. Specifically, we consider four types of information: feminine, masculine, gender-neutral and stereotypical, which represent the relationship between gender vs. bias, and propose a debiasing method that (a) preserves the genderrelated information in feminine and masculine words, (b) preserves the neutrality in genderneutral words, and (c) removes the biases from stereotypical words. Experimental results on several previously proposed benchmark datasets show that our proposed method can debias pre-trained word embeddings better than existing SoTA methods proposed for debiasing word embeddings while preserving gender-related but non-discriminative information.

show abstract

“…Indeed, using such unsupervised evaluations based on cosine similarity and Pearson correlation (which we denote UCP) may be misleading, as pointed out by Lu et al (2015). When they normalized word embeddings, their WS353 semantic similarity (Finkelstein et al, 2001) scores using UCP increased by almost 20 percentage points (pp). Since normalization is a simple operation that could easily be learned by a machine learning model, this indicates that UCP scores may yield unreliable conclusions regarding the quality of the underlying embeddings.…”

Section: Sentence Encodermentioning

confidence: 99%

Pitfalls in the Evaluation of Sentence Embeddings

Eger¹,

Rücklé²,

Gurevych³

2019

Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)

View full text Add to dashboard Cite

Deep learning models continuously break new records across different NLP tasks. At the same time, their success exposes weaknesses of model evaluation. Here, we compile several key pitfalls of evaluation of sentence embeddings, a currently very popular NLP paradigm. These pitfalls include the comparison of embeddings of different sizes, normalization of embeddings, and the low (and diverging) correlations between transfer and probing tasks. Our motivation is to challenge the current evaluation of sentence embeddings and to provide an easy-to-access reference for future research. Based on our insights, we also recommend better practices for better future evaluations of sentence embeddings.

show abstract

Placing search in context

Cited by 604 publications

References 6 publications

Correlation Coefficients and Semantic Textual Similarity

Correlation Coefficients and Semantic Textual Similarity

Gender-preserving Debiasing for Pre-trained Word Embeddings

Pitfalls in the Evaluation of Sentence Embeddings

Contact Info

Product

Resources

About