Modeling Word Meaning in Context with Substitute Vectors

Abstract

Context representations are a key element in distributional models of word meaning. In contrast to typical representations based on neighboring words, a recently proposed approach suggests to represent a context of a target word by a substitute vector, comprising the potential fillers for the target word slot in that context. In this work we first propose a variant of substitute vectors, which we find particularly suitable for measuring context similarity. Then, we propose a novel model for representing word meaning in context based on this context representation. Our model outperforms state-of-the-art results on lexical substitution tasks in an unsupervised setting.

Language Transfer Learning for Supervised Lexical Substitution

“…As optimization metric we have explored both NDCG@10 and MAP. The NDCG metric can incorporate different scoring weights 8 The omission of MWE by multiple authors has been confirmed by the authors of (Melamud et al, 2015a).…”

“…We adapt the unsupervised approach by (Melamud et al, 2015a) as a set of features. We follow (Levy and Goldberg, 2014) to construct dependency-based word embeddings; we obtain syntactic contexts by running a syntactic dependency parser 6 , and computing word embeddings using dependency edges as context features 7 .…”

“…a dependency edge attached to t; let v t , v s be the vector representations of t and s in the word embedding space, and v c the vector representation of c in the context embedding space. Then Sim 1 = cos (v s , v c ) and Sim 2 = cos (v s , v t ) are the first-order and second-order substitutability measures considered by Melamud et al (2015a). In contrast to their approach, we do not just consider an unsupervised combination of these two measures, but instead use both Sim 1 and Sim 2 as single features.…”

Sense Contextualization in a Dependency-Based Compositional Distributional Model

“…So, they cannot be merged. To combine them, on the basis of previous work (Thater et al, 2010;Erk and Padó, 2008;Melamud et al, 2015), we distinguish between direct denotation and selectional preferences within a dependency relation. Our approach is an attempt to join the main ideas of these syntax-based and structured vector space models into an entirely compositional model.…”

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

“…From the set of assertions extracted in §3.2, we create a dataset of relation-argument pairs, and use word2vecf to train the embeddings. We also tried to use the arguments' embeddings to induce a contextsensitive measure of similarity, as suggested by Melamud et al (2015); however, this method did not improve performance on our dataset.…”