A critique of word similarity as a method for evaluating
            distributional semantic models

Batchkarov, Miroslav; Kober, Thomas; Reffin, Jeremy; Weeds, Julie; Weir, David

doi:10.18653/v1/w16-2502

Cited by 44 publications

(44 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1 again). Finally, unlike with SimLex-999 or MEN scores where it is difficult to interpret "what a similarity/relatedness of 7.69 exactly means" (Batchkarov et al, 2016;Avraham and Goldberg, 2016), the USF FSG scores have a direct meaningful interpretation (i.e., F SG = #P/#G). To fully capture all aspects of the ground truth USF data set, an evaluation protocol should ideally be based not only on response rankings, but also on the actual scores, i.e., the association strength.…”

Section: Introductionmentioning

confidence: 99%

Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation

Vulić

Korhonen

2017

Proceedings of the 15th Conference of the European Chapter of The Association for Computational Linguistics: Volume 1

View full text Add to dashboard Cite

Recent work on evaluating representation learning architectures in NLP has established a need for evaluation protocols based on subconscious cognitive measures rather than manually tailored intrinsic similarity and relatedness tasks. In this work, we propose a novel evaluation framework that enables large-scale evaluation of such architectures in the free word association (WA) task, which is firmly grounded in cognitive theories of human semantic representation. This evaluation is facilitated by the existence of large manually constructed repositories of word association data. In this paper, we (1) present a detailed analysis of the new quantitative WA evaluation protocol, (2) suggest new evaluation metrics for the WA task inspired by its direct analogy with information retrieval problems, (3) evaluate various state-of-the-art representation models on this task, and (4) discuss the relationship between WA and prior evaluations of semantic representation with well-known similarity and relatedness evaluation sets. We have made the WA evaluation toolkit publicly available.

show abstract

Section: Introductionmentioning

confidence: 99%

Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation

Vulić

Korhonen

2017

Proceedings of the 15th Conference of the European Chapter of The Association for Computational Linguistics: Volume 1

View full text Add to dashboard Cite

show abstract

“…Besides the identification of each LKB, the relation set and the algorithm used are revealed for each result, plus the mean Spearman correlation (ρ) and the standard deviation (σ). The latter were computed as suggested by Batchkarov et al [42], who criticise how word similarity tests are used for assessing similarity models. What happens is that even the largest test, RareWords, is too small for taking conclusions about the performance of a broad-coverage resource, that aims at covering the whole language.…”

Section: Results For Lkbsmentioning

confidence: 99%

Distributional and Knowledge-Based Approaches for Computing Portuguese Word Similarity

Oliveira

2018

Information

View full text Add to dashboard Cite

Abstract:Identifying similar and related words is not only key in natural language understanding but also a suitable task for assessing the quality of computational resources that organise words and meanings of a language, compiled by different means. This paper, which aims to be a reference for those interested in computing word similarity in Portuguese, presents several approaches for this task and is motivated by the recent availability of state-of-the-art distributional models of Portuguese words, which add to several lexical knowledge bases (LKBs) for this language, available for a longer time. The previous resources were exploited to answer word similarity tests, which also became recently available for Portuguese. We conclude that there are several valid approaches for this task, but not one that outperforms all the others in every single test. Distributional models seem to capture relatedness better, while LKBs are better suited for computing genuine similarity, but, in general, better results are obtained when knowledge from different sources is combined.

show abstract

“…Recently, the use of word similarity methods has been criticised as a reliable technique for evaluating distributional semantic models (Batchkarov et al, 2016), given the small size of the datasets and the limitation of context information as well. However, given this procedure still is widely accepted, we have performed two different kinds of experiments: rating by similarity and synonym detection with multiple-choice questions.…”

Section: Word Similaritymentioning

confidence: 99%

Compositional Semantics using Feature-Based Models from WordNet

Gamallo¹,

Pereira-Fariña²

2017

Proceedings of the 1st Workshop on Sense, Concept and Entity Representations and Their Applications

View full text Add to dashboard Cite

This article describes a method to build semantic representations of composite expressions in a compositional way by using WordNet relations to represent the meaning of words. The meaning of a target word is modelled as a vector in which its semantically related words are assigned weights according to both the type of the relationship and the distance to the target word. Word vectors are compositionally combined by syntactic dependencies. Each syntactic dependency triggers two complementary compositional functions: the named head function and dependent function. The experiments show that the proposed compositional method performs as the state-of-the-art for subjectverb expressions, and clearly outperforms the best system for transitive subject-verbobject constructions.

show abstract

A critique of word similarity as a method for evaluating distributional semantic models

Cited by 44 publications

References 17 publications

Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation

Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation

Distributional and Knowledge-Based Approaches for Computing Portuguese Word Similarity

Compositional Semantics using Feature-Based Models from WordNet

Contact Info

Product

Resources

About