Improving Lexical Embeddings with Semantic Knowledge

Abstract

Word embeddings learned on unlabeled data are a popular tool in semantics, but may not capture the desired semantics. We propose a new learning objective that incorporates both a neural language model objective (Mikolov et al., 2013) and prior knowledge from semantic resources to learn improved lexical semantic embeddings. We demonstrate that our embeddings improve over those learned solely on raw text in three settings: language modeling, measuring semantic similarity, and predicting human judgements.

Dict2vec : Learning Word Embeddings using Lexical Dictionaries

“…Lexical databases like WordNet or sets of synonyms like MyThes thesaurus can be used during learning or in a post-processing step to specialize word embeddings. For example, Yu and Dredze (2014) include prior knowledge about synonyms from WordNet and the Paraphrase Database in a joint model built upon Word2vec. Faruqui et al (2015) introduce a graph-based retrofitting method where they post-process learned vectors with respect to semantic relationships extracted from additional lexical resources.…”

“…Recent approaches have proposed to tackle this issue using an attentive model for context selection (Ling et al, 2015), or by using external sources -like knowledge graphsin order to improve the embeddings . Similarities derived from such resources are part of the objective function during the learning phase (Yu and Dredze, 2014;Kiela et al, 2015) or used in a retrofitting scheme (Faruqui et al, 2015). These approaches tend to specialize the embeddings to the resource used and its associated similarity measures -while the construction and maintenance of these resources are a set of complex, time-consuming, and error-prone tasks.…”

Dual Tensor Model for Detecting Asymmetric Lexico-Semantic Relations

“…However, we often need embeddings to be similar only if an exact lexico-semantic relation holds between the words. Numerous methods for specializing word embeddings for particular relations have been proposed (Yu and Dredze, 2014;Faruqui et al, 2015;Kiela et al, 2015;Mrkšić et al, 2016, inter alia), primarily aiming to differentiate synonymic similarity from other types of semantic relatedness.…”

“…Yu and Dredze (2014) extend the CBOW objective with synonymy constraints from WordNet and Paraphrase Database (PPDB) (Ganitkevitch et al, 2013). Similarly, Kiela et al (2015) add synonyms as additional contexts for the skip-gram objective.…”

“…Consequently, a number of approaches have been proposed for specializing distributional spaces for specific lexico-semantic relations, either by (1) modifying the learning objective or regularization of the original embedding model by incorporating linguistic constraints (Yu and Dredze, 2014;Kiela et al, 2015) or (2) retroactively fitting the pre-trained unspecialized embeddings to linguistic constraints (Faruqui et al, 2015;Mrkšić et al, 2016). However, these methods specialize distributional vector spaces primarily for detecting the symmetric relation of semantic similarity (i.e., graded synonymy) and not for asymmetric lexico-semantic relations such as hypernymy and meronymy.…”