End-to-End Neural Entity Linking

Kolitsas, Nikolaos; Ganea, Octavian-Eugen; Hofmann, Thomas

doi:10.48550/arxiv.1808.07699

Cited by 17 publications

(25 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We experiment with two settings zero-shot assuming we know the location of the entities and the full fine-tuning setting following the exact methodology proposed in De Cao et al (2020). Specifically for the end-to-end Entity Linking, we aim to reproduce the setting of Kolitsas et al (2018). We evaluate using the aforementioned InKB micro-F 1 with the same defined in-domain and out-of-domain datasets as described by De Cao et al…”

Section: Entity Linkingmentioning

confidence: 99%

CM3: A Causal Masked Multimodal Model of the Internet

Aghajanyan¹,

Bernie²,

Ross³

et al. 2022

Preprint

View full text Add to dashboard Cite

We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents that can contain both text and image tokens. Our new causally masked approach generates tokens left to right while also masking out a small number of long token spans that are generated at the end of the string, instead of their original positions. The casual masking object provides a type of hybrid of the more common causal and masked language models, by enabling full generative modeling while also providing bidirectional context when generating the masked spans. We train causally masked languageimage models on large-scale web and Wikipedia articles, where each document contains all of the text, hypertext markup, hyperlinks, and image tokens (from a VQVAE-GAN), provided in the order they appear in the original HTML source (before masking). The resulting CM3 models can generate rich structured, multimodal outputs while conditioning on arbitrary masked document contexts, and thereby implicitly learn a wide range of text, image, and cross modal tasks. They can be prompted to recover, in a zero-shot fashion, the functionality of models such as DALL-E, GENRE, and HTLM De Cao et al., 2020;Aghajanyan et al., 2021). We set the new state-of-the-art in zero-shot summarization, entity linking, and entity disambiguation while maintaining competitive performance in the fine-tuning setting. We can generate images unconditionally, conditioned on text (like DALL-E) and do captioning all in a zero-shot setting with a single model.

show abstract

Section: Entity Linkingmentioning

confidence: 99%

CM3: A Causal Masked Multimodal Model of the Internet

Aghajanyan¹,

Bernie²,

Ross³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We plan to identify more accurate entities by relying on attention weights in LMs (Clark et al, 2019;Hewitt & Manning, 2019) instead of using extra resources. We will also investigate stronger entity linkers (Kolitsas et al, 2018) and learn a more robust relation mapping through weak or distant supervision (Mintz et al, 2009;Ratner et al, 2017). We will investigate more sophisticated approaches, such as graph neural networks (Kipf & Welling, 2016), to generate more accurate relation phrases from the attention weight matrices by considering structural information.…”

Section: Analysis Of Unmapped Factsmentioning

confidence: 99%

Language Models are Open Knowledge Graphs

Wang,

Liu,

Song

2020

Preprint

View full text Add to dashboard Cite

This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the pretrained language models (without fine-tuning) over the corpora. We demonstrate the quality of the constructed KGs by comparing to two KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual knowledge that is new in the existing KGs. Our code and KGs will be made publicly available.

show abstract

“…So in this study, we will explore a different approach, which initializes larger position embeddings based on the existing small one in BERT-Base, and can be used directly in the finetuning without expensive retraining. 2019), we adopt a two-stage pipeline consisting of a fast candidate generation stage, followed by a more expensive but powerful candidate ranking stage (Ganea and Hofmann, 2017;Kolitsas et al, 2018;Wu et al, 2019). We use BM25 for the candidate generation stage and get 64 candidate entities for every mention.…”

Section: Modeling Long Documentsmentioning

confidence: 99%

“…Figure 2 describes how to use BERT for zeroshot entity linking tasks with larger position embeddings. Following Logeswaran et al (2019), we adopt a two-stage pipeline consisting of a fast candidate generation stage, followed by a more expensive but powerful candidate ranking stage(Ganea and Hofmann, 2017;Kolitsas et al, 2018;Wu et al, 2019). We use BM25 for the candidate generation stage and get 64 candidate entities for every mention.…”

mentioning

confidence: 99%

Zero-shot Entity Linking with Efficient Long Range Sequence Modeling

Yao

Cao

Pan

2020

Preprint

View full text Add to dashboard Cite

This paper considers the problem of zero-shot entity linking, in which a link in the test time may not present in training. Following the prevailing BERT-based research efforts, we find a simple yet effective way is to expand the long-range sequence modeling. Unlike many previous methods, our method does not require expensive pre-training of BERT with long position embeddings. Instead, we propose an efficient position embeddings initialization method called Embedding-repeat, which initializes larger position embeddings based on BERT-Base. On Wikia's zero-shot EL dataset, our method improves the SOTA from 76.06% to 79.08%, and for its long data, the corresponding improvement is from 74.57% to 82.14%. Our experiments suggest the effectiveness of long-range sequence modeling without retraining the BERT model. 1

show abstract

End-to-End Neural Entity Linking

Cited by 17 publications

References 0 publications

CM3: A Causal Masked Multimodal Model of the Internet

CM3: A Causal Masked Multimodal Model of the Internet

Language Models are Open Knowledge Graphs

Zero-shot Entity Linking with Efficient Long Range Sequence Modeling

Contact Info

Product

Resources

About