DeepType: Multilingual Entity Linking by Neural Type System Evolution

Abstract

The wealth of structured (e.g. Wikidata) and unstructured data about the world available today presents an incredible opportunity for tomorrow's Artificial Intelligence. So far, integration of these two different modalities is a difficult process, involving many decisions concerning how best to represent the information so that it will be captured or useful, and hand-labeling large amounts of data.DeepType overcomes this challenge by explicitly integrating symbolic information into the reasoning process of a neural network with a type system.First we construct a type system, and second, we use it to constrain the outputs of a neural network to respect the symbolic structure. We achieve this by reformulating the design problem into a mixed integer problem: create a type system and subsequently train a neural network with it. In this reformulation discrete variables select which parent-child relations from an ontology are types within the type system, while continuous variables control a classifier fit to the type system. The original problem cannot be solved exactly, so we propose a 2-step algorithm: 1) heuristic search or stochastic optimization over discrete variables that define a type system informed by an Oracle and a Learnability heuristic, 2) gradient descent to fit classifier parameters.We apply DeepType to the problem of Entity Linking on three standard datasets (i.e. WikiDisamb30, CoNLL (YAGO), TAC KBP 2010) and find that it outperforms all existing solutions by a wide margin, including approaches that rely on a human-designed type system or recent deep learning-based entity embeddings, while explicitly using symbolic information lets it integrate new entities without retraining.

Growing and Serving Large Open-domain Knowledge Graphs

“…Due to the significant growth in the size of KGs in recent years, there is an active line of research on designing systems that can handle large-scale training of KG embeddings, such as Marius [16,19], Pytorch Biggraph [15], and DGL-KE [24]. Various semantic annotation tasks, including entity linking and disambiguation, have received renewed interest from the research community, especially for deep learning based approaches [8,13,14,17]. Prior works have also discussed the applications of semantic annotations to enrich webpages for improved question answering and web search [1].…”

Doc2KG

“…However, the proposed frameworks did not deal with entity linking to existing domain standards for semantic interoperability. Other methods focusing on entity linking considered the existence of a knowledge graph without dealing with the cold start problem by constructing a knowledge graph from the beginning, where there is no available information on an instance level (Ganea & Hofmann, 2017;Raiman & Raiman, 2018). Towards this regard, Rossanez et al (2020) proposed an approach for knowledge graph construction from biomedical scientific articles.…”

Improving broad-coverage medical entity linking with semantic type prediction and large-scale datasets

“…Knowledge bases often group entities into semantic types, which inform several downstream NLP tasks such as coreference resolution [29], relation extraction [30], question answering [31], and language modeling [32]. Recent studies have shown that finegrained entity type prediction improves entity linking in Wikipedia text [33,22], indicating a clear potential for type prediction as a standard component of entity linking pipelines.…”

“…AttentionNER [77] utilizes attention mechanism for extracting relevant information from the context of a mention for type prediction. DeepType-FC and DeepType-RNN are two neural network based models proposed by [22] for entity typing. Type-CNN [78] is another neural approach which utilizes CNNs for modeling the global context of a mention for type prediction.…”

“…Identifying the semantic type of mentions has previously been shown to improve entity linking performance in Wikipedia [22]. However, this idea has not yet been systematically explored for medical entity linking, in part due to the dearth of annotated training data for the task.…”