Zero-shot Fact Verification by Claim Generation

Abstract

Neural models for automated fact verification have achieved promising results thanks to the availability of large, human-annotated datasets. However, for each new domain that requires fact verification, creating a dataset by manually writing claims and linking them to their supporting evidence is expensive. We develop QACG, a framework for training a robust fact verification model by using automaticallygenerated claims that can be supported, refuted, or unverifiable from evidence from Wikipedia. QACG generates question-answer pairs from the evidence and then convert them into different types of claims. Experiments on the FEVER dataset show that our QACG framework significantly reduces the demand for human-annotated training data. In a zeroshot scenario, QACG improves a RoBERTa model's F 1 from 50% to 77%, equivalent in performance to 2K+ manually-curated examples. Our QACG code is publicly available. 1

Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

“…Automatically generated texts can be used to generate claim-evidence pairs for each label category. To this end, Pan et al (2021a) employ a two-step approach to generate synthetic data for fact verification. In the first step of question generation, given the evidence and an answer, a BART model, fine-tuned on the SQuAD dataset using the similar input-output format, generates a question for that answer.…”

A Survey on Automated Fact-Checking

“…Recently considered using the information stored in the weights of a large pretrained language model -BERT (Devlin et al, 2019) -as the only source of evidence, as it has been shown competitive in knowledge base completion (Petroni et al, 2019). However, without explicitly considering evidence, such approaches are likely to propagate biases learned from the data they were trained on, and render justification generation impossible (Lee et al, 2021;Pan et al, 2021).…”

BiRdQA: A Bilingual Dataset for Question Answering on Tricky Riddles

“…Global Distractor For each English riddle, we use the correct answer as the query and retrieve the top-2 most similar words or phrases from the pretrained Sense2Vec (Trask, Michalak, and Liu 2015;Pan et al 2021). For each Chinese riddle, we use the word embedding provided by Song et al (2018).…”

“…Invalid Distractor Replacement To avoid the case that the generated distractor becomes another correct answer or repeats with other distractors, we define rules to ensure that the distractor has less lexical overlap with the correct answer (Pan et al 2021) and other distractors of the same riddle. Specifically, two candidates are considered as lexically overlapped if they share at least a same word in English (e.g., "horse" and "wild horse") or a same character in Chinese (e.g., "蘑菇" and "平菇").…”