Benchmarking Semantic Capabilities of Analogy Querying Algorithms

Lofi, Christoph; Ahamed, A.F.; Kulkarni, Pratima; Thakkar, Ravi

doi:10.1007/978-3-319-32025-0_29

Cited by 1 publication

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to those already cited, numerous other recent papers have evaluated word embeddings by benchmarking on analogy questions (Mikolov et al, 2013b;Garten et al, 2015;Lofi et al, 2016). There is some consensus regarding performance across question types: systems do well on questions of inflectional morphology (especially so for English (Nicolai et al, 2015)), but far less reliably so for various non-geographical semantic questions-although some gains in performance are possible by adjusting the embedding algorithms used or their hyperparameters (Levy et al, 2015), or by training further on subproblems (Drozd et al, 2016).…”

Section: Accounting For Analogy Performancementioning

confidence: 99%

What Analogies Reveal about Word Vectors and their Compositionality

Finley¹,

Farmer²,

Pakhomov³

2017

Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017)

View full text Add to dashboard Cite

Analogy completion via vector arithmetic has become a common means of demonstrating the compositionality of word embeddings. Previous work have shown that this strategy works more reliably for certain types of analogical word relationships than for others, but these studies have not offered a convincing account for why this is the case. We arrive at such an account through an experiment that targets a wide variety of analogy questions and defines a baseline condition to more accurately measure the efficacy of our system. We find that the most reliably solvable analogy categories involve either 1) the application of a morpheme with clear syntactic effects, 2) male-female alternations, or 3) named entities. These broader types do not pattern cleanly along a syntacticsemantic divide. We suggest instead that their commonality is distributional, in that the difference between the distributions of two words in any given pair encompasses a relatively small number of word types. Our study offers a needed explanation for why analogy tests succeed and fail where they do and provides nuanced insight into the relationship between word distributions and the theoretical linguistic domains of syntax and semantics.

show abstract