SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

Hill, Felix; Reichart, Roi; Korhonen, Anna

doi:10.1162/coli_a_00237

Cited by 997 publications

(1,068 citation statements)

References 54 publications

Supporting

Mentioning

1,053

Contrasting

Unclassified

Order By: Relevance

“…1 This bag-of-words approach, however, comes with a cost. As recently shown by Hill et al (2014), despite the impressive results VSMs that take this approach obtain on modeling word association, they are much less successful in modeling word similarity. Indeed, when evaluating these VSMs with datasets such as wordsim353 (Finkelstein et al, 2001), where the word pair scores re-flect association rather than similarity (and therefore the (cup,coffee) pair is scored higher than the (car,train) pair), the Spearman correlation between their scores and the human scores often crosses the 0.7 level.…”

Section: Introductionmentioning

confidence: 99%

“…Indeed, when evaluating these VSMs with datasets such as wordsim353 (Finkelstein et al, 2001), where the word pair scores re-flect association rather than similarity (and therefore the (cup,coffee) pair is scored higher than the (car,train) pair), the Spearman correlation between their scores and the human scores often crosses the 0.7 level. However, when evaluating with datasets such as SimLex999 (Hill et al, 2014), where the pair scores reflect similarity, the correlation of these models with human judgment is below 0.5 (Section 6).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction

Schwartz¹,

Reichart²,

Rappoport³

2015

Proceedings of the Nineteenth Conference on Computational Natural Language Learning

Self Cite

100

View full text Add to dashboard Cite

We present a novel word level vector representation based on symmetric patterns (SPs). For this aim we automatically acquire SPs (e.g., "X and Y") from a large corpus of plain text, and generate vectors where each coordinate represents the cooccurrence in SPs of the represented word with another word of the vocabulary. Our representation has three advantages over existing alternatives: First, being based on symmetric word relationships, it is highly suitable for word similarity prediction. Particularly, on the SimLex999 word similarity dataset, our model achieves a Spearman's ρ score of 0.517, compared to 0.462 of the state-of-the-art word2vec model. Interestingly, our model performs exceptionally well on verbs, outperforming stateof-the-art baselines by 20.2-41.5%. Second, pattern features can be adapted to the needs of a target NLP application. For example, we show that we can easily control whether the embeddings derived from SPs deem antonym pairs (e.g. (big,small)) as similar or dissimilar, an important distinction for tasks such as word classification and sentiment analysis. Finally, we show that a simple combination of the word similarity scores generated by our method and by word2vec results in a superior predictive power over that of each individual model, scoring as high as 0.563 in Spearman's ρ on SimLex999. This emphasizes the differences between the signals captured by each of the models.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction

Schwartz¹,

Reichart²,

Rappoport³

2015

Proceedings of the Nineteenth Conference on Computational Natural Language Learning

Self Cite

100

View full text Add to dashboard Cite

show abstract

“…For word similarity evaluations, we use the WordSim-353 Similarity (WS-Sim) and Relatedness (WS-Rel) (Finkelstein et al, 2001) and SimLex-999 (SimLex) (Hill et al, 2015) datasets, and the Rare Word (RW) (Luong et al, 2013) dataset to verify if subword information improves rare word representation. Relationships are measured using the Google semantic (GSem) and syntactic (GSyn) analogies (Mikolov et al, 2013a) and the Microsoft syntactic analogies (MSR) dataset (Mikolov et al, 2013b).…”

Section: Methodsmentioning

confidence: 99%

Proceedings of the Second Workshop on Subword/Character LEvel Models

2018

View full text Add to dashboard Cite

ii Introduction Traditional NLP starts with a hand-engineered layer of representation, the level of tokens or words. A tokenization component first breaks up the text into units using manually designed rules. Tokens are then processed by components such as word segmentation, morphological analysis and multiword recognition. The heterogeneity of these components makes it hard to create integrated models of both structure within tokens (e.g., morphology) and structure across multiple tokens (e.g., multi-word expressions). This approach can perform poorly (i) for morphologically rich languages, (ii) for noisy text, (iii) for languages in which the recognition of words is difficult and (iv) for adaptation to new domains; and (v) it can impede the optimization of preprocessing in end-to-end learning.The workshop provides a forum for discussing recent advances as well as future directions on sub-word and character-level natural language processing and representation learning that address these problems. Topics of Interest:• tokenization-free models• character-level machine translation• character-ngram information retrieval• transfer learning for character-level models• models of within-token and cross-token structure• NL generation (of words not seen in training etc)• out of vocabulary words• morphology and segmentation• relationship b/w morphology and character-level models• stemming and lemmatization • inflection generation• orthographic productivity• form-meaning representations• true end-to-end learning AbstractNeural machine translation has achieved impressive results in the last few years, but its success has been limited to settings with large amounts of parallel data. One way to improve NMT for lower-resource settings is to initialize a word-based NMT model with pretrained word embeddings. However, rare words still suffer from lower quality word embeddings when trained with standard word-level objectives. We introduce word embeddings that utilize morphological resources, and compare to purely unsupervised alternatives. We work with Arabic, a morphologically rich language with available linguistic resources, and perform Ar-to-En MT experiments on a small corpus of TED subtitles. We find that word embeddings utilizing subword information consistently outperform standard word embeddings on a word similarity task and as initialization of the source word embeddings in a low-resource NMT system.

show abstract

“…They are generally used to evaluate computational models of similarity (Faruqui and Dyer 2014). However, since most of these collections are limited to pairs of nouns, with the exception of Hill et al (2015), Yang and Powers (2006) and Baker et al (2014), the coverage of verbs (and other categories) is reduced. For Spanish there is a general scarcity of these resources: there are only ratings for nouns, either collected from native speakers (Moldovan 2015), or translated from English (Finkelstein et al 2001;Camacho-Collados et al 2015).…”

Section: Background On Similaritymentioning

confidence: 99%

Verb similarity: Comparing corpus and psycholinguistic data

Gil-Vallejo

Coll-Florit

Masalles

et al. 2018

Corpus Linguistics and Linguistic Theory

View full text Add to dashboard Cite

Similarity, which plays a key role in fields like cognitive science, psycholinguistics and natural language processing, is a broad and multifaceted concept. In this work we analyse how two approaches that belong to different perspectives, the corpus view and the psycholinguistic view, articulate similarity between verb senses in Spanish. Specifically, we compare the similarity between verb senses based on their argument structure, which is captured through semantic roles, with their similarity defined by word associations. We address the question of whether verb argument structure, which reflects the expression of the events, and word associations, which are related to the speakers' organization of the mental lexicon, shape similarity between verbs in a congruent manner, a topic which has not been explored previously. While we find significant correlations between verb sense similarities obtained from these two approaches, our findings also highlight some discrepancies between them and the importance of the degree of abstraction of the corpus annotation and psycholinguistic representations.

show abstract

SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

Cited by 997 publications

References 54 publications

Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction

Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction

Proceedings of the Second Workshop on Subword/Character LEvel Models

Verb similarity: Comparing corpus and psycholinguistic data

Contact Info

Product

Resources

About