Learning Dense Representations for Entity Retrieval

Abstract

We show that it is feasible to perform entity linking by training a dual encoder (two-tower) model that encodes mentions and entities in the same dense vector space, where candidate entities are retrieved by approximate nearest neighbor search. Unlike prior work, this setup does not rely on an alias table followed by a re-ranker, and is thus the first fully learned entity retrieval model. We show that our dual encoder, trained using only anchor-text links in Wikipedia, outperforms discrete alias table and BM25 baselines, and is competitive with the best comparable results on the standard TACKBP-2010 dataset. In addition, it can retrieve candidates extremely fast, and generalizes well to a new dataset derived from Wikinews. On the modeling side, we demonstrate the dramatic value of an unsupervised negative mining algorithm for this task.

Entity Linking in 100 Languages

“…Basing entity representations on features of their Wikipedia pages has been a common approach in EL (e.g. Sil and Florian, 2016;Francis-Landau et al, 2016;Gillick et al, 2019;Wu et al, 2019), but we will need to generalize this to include multiple Wikipedia pages with possibly redundant features in many languages.…”

“…Prior work showed that a dual encoder architecture can encode entities and contextual mentions in a dense vector space to facilitate efficient entity retrieval via nearest-neighbors search (Gillick et al, 2019;Wu et al, 2019). We take the same approach.…”

“…Training with hard-negatives is highly effective in monolingual entity retrieval (Gillick et al, 2019), and we apply the technique they detail to our multilingual setting. Figure B1 in the Appendix for a larger view.…”

Learning Contextualized Document Representations for Healthcare Answer Retrieval

“…In contrast to interaction-based models, which are applied to query-document pairs, our approach is to decouple entity encoding and document encoding. Therefore we follow recent work in representation-based Entity Linking [14] and embed textual knowledge from the clinical domain into this representation. Our goal is to generalize entity representations, so the model will be able to align to existing taxonomies without retraining.…”

“…The model uses multi-task learning [9] to align the sequence of sentences in a long document to the clinical knowledge encoded in pre-trained entity and aspect vector spaces. We use 1 Code and evaluation data is available at https://github.com/sebastianarnold/cdv a dual encoder architecture [15], which allows us to precompute discourse vectors for all documents and later answer ad-hoc queries over that corpus with short latency [14]. Consequently, the model predicts similarity scores with sentence granularity and does not require an extra inference step after the initial document indexing.…”