Vector Space Models of Word Meaning and Phrase Meaning: A Survey

Abstract

Distributional models represent a word through the contexts in which it has been observed. They can be used to predict similarity in meaning, based on the distributional hypothesis, which states that two words that occur in similar contexts tend to have similar meanings. Distributional approaches are often implemented in vector space models. They represent a word as a point in high‐dimensional space, where each dimension stands for a context item, and a word's coordinates represent its context counts. Occurrence in similar contexts then means proximity in space. In this survey we look at the use of vector space models to describe the meaning of words and phrases: the phenomena that vector space models address, and the techniques that they use to do so. Many word meaning phenomena can be described in terms of semantic similarity: synonymy, priming, categorization, and the typicality of a predicate's arguments. But vector space models can do more than just predict semantic similarity. They are a very flexible tool, because they can make use of all of linear algebra, with all its data structures and operations. The dimensions of a vector space can stand for many things: context words, or non‐linguistic context like images, or properties of a concept. And vector space models can use matrices or higher‐order arrays instead of vectors for representing more complex relationships. Polysemy is a tough problem for distributional approaches, as a representation that is learned from all of a word's contexts will conflate the different senses of the word. It can be addressed, using either clustering or vector combination techniques. Finally, we look at vector space models for phrases, which are usually constructed by combining word vectors. Vector space models for phrases can predict phrase similarity, and some argue that they can form the basis for a general‐purpose representation framework for natural language semantics.

Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction

“…The feature representation generated by this basic construction is sometimes post-processed using techniques such as Positive Pointwise Mutual Information (PPMI) normalization and dimensionality reduction. For recent surveys, see (Turney et al, 2010;Clark, 2012;Erk, 2012).…”

Conceptual Versus Referential Affordance in Concept Composition

“…However, as it is beyond the scope of this paper to exhaustively compare these different approaches, we limit ourselves here to simply providing enough background on distributional semantics for the reader to be able to follow the formalization presented in the next section, leaving more thorough discussion for future work. Distributional semantic analyses (Landauer and Dumais 1997;Turney and Pantel 2010;Erk 2012) represent the semantics of a word as a function of the contexts it occurs in. Context can be defined in various ways, but the most typical approach is to define context as the words surrounding the target word in a corpus.…”

Classification systems offer a microcosm of issues in conceptual processing: a commentary on Kemmerer (2016)

“…Another possibility is that distributed linguistic representations play a central role in classifier semantics (e.g., Baroni & Lenci, 2010;Erk, 2012;Landauer, McNamara, Dennis, & Kintsch, 2013).…”