Jointly learning word embeddings using a corpus and a knowledge base

Abstract

Methods for representing the meaning of words in vector spaces purely using the information distributed in text corpora have proved to be very valuable in various text mining and natural language processing (NLP) tasks. However, these methods still disregard the valuable semantic relational structure between words in co-occurring contexts. These beneficial semantic relational structures are contained in manually-created knowledge bases (KBs) such as ontologies and semantic lexicons, where the meanings of words are represented by defining the various relationships that exist among those words. We combine the knowledge in both a corpus and a KB to learn better word embeddings. Specifically, we propose a joint word representation learning method that uses the knowledge in the KBs, and simultaneously predicts the co-occurrences of two words in a corpus context. In particular, we use the corpus to define our objective function subject to the relational constrains derived from the KB. We further utilise the corpus co-occurrence statistics to propose two novel approaches, Nearest Neighbour Expansion (NNE) and Hedged Nearest Neighbour Expansion (HNE), that dynamically expand the KB and therefore derive more constraints that guide the optimisation process. Our experimental results over a wide-range of benchmark tasks demonstrate that the proposed method statistically significantly improves the accuracy of the word embeddings learnt. It outperforms a corpus-only baseline and reports an improvement of a number of previously proposed methods that incorporate corpora and KBs in both semantic similarity prediction and word analogy detection tasks.

A Year of Papers Using Biomedical Texts: Findings from the Section on Natural Language Processing of the IMIA Yearbook

“…Besides, word embeddings are exploited in a great variety of works, such as for the detection of novel and emerging drug terms 19 or for the simplification 11 . It was also noted that the quality of word embeddings can be improved through the combination of corpora and knowledge bases 24 .…”

Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks

“…• Counter-fit (Mrkšić et al, 2016), a method for injecting both antonym and synonym constraints into word embeddings. • JointReps (Alsuhaibani et al, 2018), a joint word representation learning method which simultaneously utilizes the corpus and KB.…”

“…Incorporating relevant signals from semantic knowledge sources such as WordNet (Miller, 1995), FrameNet (Baker et al, 1998), and Paraphrase Database (PPDB) (Pavlick et al, 2015) has been shown to improve the quality of word embeddings. Recent works utilize these by incorporating them in a neural language modeling objective function (Yu and Dredze, 2014;Alsuhaibani et al, 2018), or as a post-processing step (Faruqui et al, 2014;Mrkšić et al, 2016). Although existing approaches improve the quality of word embeddings, they require explicit modification for handling different types of semantic information.…”

Understanding the Semantic Content of Sparse Word Embeddings Using a Commonsense Knowledge Base

“…Osborne et al (2016) introduced an algorithm for determining word representations that also encode prior knowledge into the learned embeddings besides the distributional information originating from raw text corpora. Alsuhaibani et al (2018) consider a learning process where a word embedding and a knowledge base are learned together. The knowledge base is incorporated into the embedding in an implicit manner by integrating it into the objective function (i.e., vectors of words being in a relation are supposed to be close).…”