End-to-End Entity Resolution and Question Answering Using Differentiable Knowledge Graphs

Oliya, Armin; Saffari, Amir; Sen, P.K.; Ayoola, Tom

doi:10.18653/v1/2021.emnlp-main.345

Cited by 12 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For datasets, we use WebQSP (Yih et al, 2016) that is modified from Berant et al (2013) to filter out unanswerable questions, and Mintaka (Sen et al, 2022). Further, for the knowledge source, we use Wikidata which includes billions of facts that are represented as the triplet: (subject, relation, object), and we follow the standard preprocessing setup for KGQA (Saffari et al, 2021;Baek et al, 2023).…”

Section: Open-domain Question Answeringmentioning

confidence: 99%

Knowledge-Augmented Language Model Verification

Baek,

Jeong,

Kang

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Recent Language Models (LMs) have shown impressive capabilities in generating texts with the knowledge internalized in parameters. Yet, LMs often generate the factually incorrect responses to the given queries, since their knowledge may be inaccurate, incomplete, and outdated. To address this problem, previous works propose to augment LMs with the knowledge retrieved from an external knowledge source. However, such approaches often show suboptimal text generation performance due to two reasons: 1) the model may fail to retrieve the knowledge relevant to the given query, or 2) the model may not faithfully reflect the retrieved knowledge in the generated text. To overcome these, we propose to verify the output and the knowledge of the knowledge-augmented LMs with a separate verifier, which is a small LM that is trained to detect those two types of errors through instruction-finetuning. Then, when the verifier recognizes an error, we can rectify it by either retrieving new knowledge or generating new text. Further, we use an ensemble of the outputs from different instructions with a single verifier to enhance the reliability of the verification processes. We validate the effectiveness of the proposed verification steps on multiple question answering benchmarks, whose results show that the proposed verifier effectively identifies retrieval and generation errors, allowing LMs to provide more factually correct outputs. Our code is available at https://github.com/JinheonBaek/KALMV.

show abstract

Section: Open-domain Question Answeringmentioning

confidence: 99%

Knowledge-Augmented Language Model Verification

Baek,

Jeong,

Kang

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…These QA systems are designed to answer questions from datasets such as Natural Questions (Kwiatkowski et al, 2019). The knowledge needed to answer questions can be in pre-trained models , knowledge-graphs (KGs) (Lin et al, 2019;Feng et al, 2020;Lv et al, 2020;Saffari et al, 2021) or document collections (Chen et al, 2017;Izacard and Grave, 2021;Guu et al, 2020;. In retrieval-based systems, differential retrieval can be combined with extractive question answering, as in REALM (Guu et al, 2020) and ORQA , as well as with generative answer generation, as in RAG .…”

Section: Related Workmentioning

confidence: 99%

Retrieval Augmented Visual Question Answering with Outside Knowledge

Lin¹,

Byrne²

2022

Preprint

View full text Add to dashboard Cite

Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to answer questions about images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve documents from external knowledge bases, such as Wikipedia, but with DPR trained separately from answer generation, introducing a potential limit on the overall system performance. Instead, we propose a joint training scheme which includes differentiable DPR integrated with answer generation so that the system can be trained in an end-to-end fashion. Our experiments show that our scheme outperforms recent OK-VQA systems with strong DPR for retrieval. We also introduce new diagnostic metrics to analyze how retrieval and generation interact. The strong retrieval ability of our model significantly reduces the number of retrieved documents needed in training, yielding significant benefits in answer quality and computation required for training.

show abstract

“…Hence, annotation in RE is the process of chosing relations between head and tail entities in the context from a collection of relation types. Relation Extraction is an indispensable component for composing the knowledge graphs Li et al (2019) that are useful to various downstream NLP applications such as question answering (Dubey, 2021;Saffari et al, 2021;Sen et al, 2021) and dialogue system (Liu et al, 2021b;Chaudhuri et al, 2021;Gao et al, 2021a).…”

Section: Re-annotation In Relation Extractionmentioning

confidence: 99%

Pre-trained Language Models as Re-Annotators

Chen¹

2022

Preprint

View full text Add to dashboard Cite

Annotation noise are widespread in datasets, but manually revising a flawed corpus is time-consuming and error-prone. Hence, given the prior knowledge in Pre-trained Language Models and the expected uniformity across all annotations, we attempt to reduce annotation noise in the corpus through two tasks automatically: (1) Annotation Inconsistency Detection that indicates the credibility of annotations, and (2) Annotation Error Correction that rectifies the abnormal annotations. We investigate how to acquire semantic sensitive annotation representations from Pre-trained Language Models, expecting to embed the examples with identical annotations to the mutually adjacent positions even without fine-tuning. We proposed a novel credibility score to reveal the likelihood of annotation inconsistencies based on the neighbouring consistency. Then, we fine-tune the Pre-trained Language Models based classifier with cross-validation for annotation correction. The annotation corrector is further elaborated with two approaches: (1) soft labelling by Kernel Density Estimation and (2) a novel distant-peer contrastive loss. We study the re-annotation in relation extraction and create a new manually revised dataset, Re-DocRED, for evaluating document-level re-annotation. The proposed credibility scores show promising agreement with human revisions, achieving a Binary F 1 of 93.4 and 72.5 in detecting inconsistencies on TACRED and DocRED respectively.Moreover, the neighbour-aware classifiers based on distant-peer contrastive learning and uncertain labels achieve Macro F 1 up to 66.2 and 57.8 in correcting annotations on TACRED and DocRED respectively. These improvements are not merely theoretical: Rather, automatically denoised training sets demonstrate up to 3.6% performance improvement for state-of-the-art relation extraction models, and the proposed framework is expected to be hundreds of times faster than the human re-annotators empirically.

show abstract

End-to-End Entity Resolution and Question Answering Using Differentiable Knowledge Graphs

Cited by 12 publications

References 12 publications

Knowledge-Augmented Language Model Verification

Knowledge-Augmented Language Model Verification

Retrieval Augmented Visual Question Answering with Outside Knowledge

Pre-trained Language Models as Re-Annotators

Contact Info

Product

Resources

About