Word Embeddings for Entity-Annotated Texts

Abstract

Learned vector representations of words are useful tools for many information retrieval and natural language processing tasks due to their ability to capture lexical semantics. However, while many such tasks involve or even rely on named entities as central components, popular word embedding models have so far failed to include entities as first-class citizens. While it seems intuitive that annotating named entities in the training corpus should result in more intelligent word features for downstream tasks, performance issues arise when popular embedding approaches are naïvely applied to entity annotated corpora. Not only are the resulting entity embeddings less useful than expected, but one also finds that the performance of the non-entity word embeddings degrades in comparison to those trained on the raw, unannotated corpus. In this paper, we investigate approaches to jointly train word and entity embeddings on a large corpus with automatically annotated and linked entities. We discuss two distinct approaches to the generation of such embeddings, namely the training of state-of-the-art embeddings on rawtext and annotated versions of the corpus, as well as node embeddings of a co-occurrence graph representation of the annotated corpus. We compare the performance of annotated embeddings and classical word embeddings on a variety of word similarity, analogy, and clustering evaluation tasks, and investigate their performance in entity-specific tasks. Our findings show that it takes more than training popular word embedding models on an annotated corpus to create entity embeddings with acceptable performance on common test cases. Based on these results, we discuss how and when node embeddings of the co-occurrence graph representation of the text can restore the performance.

Cross-lingual Entity Alignment with Incidental Supervision

“…Toutanova et al (2015) extract dependency paths from sentences and jointly embed them with a KG using DistMult (Yang et al, 2015) to support the relation extraction task. Several other approaches focus on jointly embedding words, entities (Yamada et al, 2017;Newman-Griffis et al, 2018;Cao et al, 2017;Almasian et al, 2019) and entity types (Gupta et al, 2017) appearing in the same textual contexts without considering relational structure of a KG. These ap-proaches are employed in monolingual NLP tasks including entity linking (Gupta et al, 2017;Cao et al, 2017), entity abstraction (Newman-Griffis et al, 2018) and factoid QA (Yamada et al, 2017).…”

Extraction and Representation of Financial Entities from Text

“…By annotating the text with named entity information before training the model, unique multi-word entries in the dictionary directly relate to known entities. Almasian et al propose such a model for entity-annotated texts [2]. Other interesting approaches build networks of co-occurring words and entities.…”

“…Auditing firms and law enforcement need to sift through massive amounts of data to gather evidence of criminal activity, often involving communication networks and documents [28]. Current computer-aided exploration tools, 2 offer a wide range of features from data ingestion, exploration, analysis, to visualization. This way, users can quickly navigate the underlying data based on extracted attributes, which would otherwise be infeasible due to the often large amount of heterogeneous data.…”

Low-Rank Subspaces for Unsupervised Entity Linking

“…. , m N } be the set of N mentions contained in D, and E be the set of entities in the reference KG G. A low-dimensional representation (embedding) can be learned for each entity by applying node representation learning techniques such as DEEPWALK (Perozzi et al, 2014) to the graph G. The entity embeddings learned by these techniques are known to be meaningful with respect to the relatedness of the entities they represent (Almasian et al, 2019). The EL task consists in finding, for each mention m ∈ M D , the entity e ∈ E to which it refers.…”

“…If the number of eigenthemes is chosen to be small, these components will constitute a good basis to approximate the dense region of the document embedding matrix. From the fundamental assumption of the existence of topical relatedness across the gold entities in a document and that such relatedness is captured by their corresponding embeddings (Almasian et al, 2019), the gold entities will form a dense region and, consequently, will define the subspace. However, this is only possible if there is no other subset of candidate entities whose relatedness is larger than that of the set of gold entities.…”