Language Models as Fact Checkers?

Lee, Nayeon; Li, Belinda Z.; Wang, Sinong; Yih, Wen-tau; Ma, Hao; Khabsa, Madian

doi:10.18653/v1/2020.fever-1.5

Cited by 32 publications

(24 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, the event of COVID-19 emerged after the release of the GPT2 pre-trained model. Second, although LMs have shown surprising ability in memorizing some knowledge, they are not perfect, as pointed out by previous works (Poerner et al, 2019;Lee et al, 2020). Therefore, we propose to incorporate evidence into the perplexity calculation by using it as a prefix of the claim.…”

Section: Evidence Conditioned Perplexitymentioning

confidence: 86%

Towards Few-shot Fact-Checking via Perplexity

Lee¹,

Bang²,

Madotto³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

Few-shot learning has drawn researchers' attention to overcome the problem of data scarcity. Recently, large pre-trained language models have shown great performance in fewshot learning for various downstream tasks, such as question answering and machine translation. Nevertheless, little exploration has been made to achieve few-shot learning for the fact-checking task. However, fact-checking is an important problem, especially when the amount of information online is growing exponentially every day. In this paper, we propose a new way of utilizing the powerful transfer learning ability of a language model via a perplexity score. The most notable strength of our methodology lies in its capability in fewshot learning. With only two training samples, our methodology can already outperform the Major Class baseline by more than an absolute 10% on the F1-Macro metric across multiple datasets. Through experiments, we empirically verify the plausibility of the rather surprising usage of the perplexity score in the context of fact-checking and highlight the strength of our few-shot methodology by comparing it to strong fine-tuning-based baseline models. Moreover, we construct and publicly release two new fact-checking datasets related to COVID-19.

show abstract

Section: Evidence Conditioned Perplexitymentioning

confidence: 86%

Towards Few-shot Fact-Checking via Perplexity

Lee¹,

Bang²,

Madotto³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

show abstract

“…MNLI-Transfer (U3) trains a BERT model for natural language inference on the MultiNLI corpus (Williams et al, 2018) and applies it for fact verification. LM as Fact Checker (Lee et al, 2020b) (U4) leverages the implicit knowledge stored in the pretrained BERT language model to verify a claim. The implementation details are given in Appendix C.…”

Section: Methodsmentioning

confidence: 99%

“…LM as Fact Checker (U4). Since there is no public available code for this model, we implement our own version following the settings described in Lee et al (2020b). We use Hugging-Face's bert-base as the language model to predict the masked named entity, and use the NLI model described in U3 as the entailment model.…”

Section: Mnli-transfer (U3)mentioning

confidence: 99%

Zero-shot Fact Verification by Claim Generation

Pan¹,

Chen²,

Xiong³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Neural models for automated fact verification have achieved promising results thanks to the availability of large, human-annotated datasets. However, for each new domain that requires fact verification, creating a dataset by manually writing claims and linking them to their supporting evidence is expensive. We develop QACG, a framework for training a robust fact verification model by using automaticallygenerated claims that can be supported, refuted, or unverifiable from evidence from Wikipedia. QACG generates question-answer pairs from the evidence and then convert them into different types of claims. Experiments on the FEVER dataset show that our QACG framework significantly reduces the demand for human-annotated training data. In a zeroshot scenario, QACG improves a RoBERTa model's F 1 from 50% to 77%, equivalent in performance to 2K+ manually-curated examples. Our QACG code is publicly available. 1

show abstract

“…We also have a 2-step pipeline, but methods used in each step is distinct from their work. Lee et al (2020) take a new approach using BERT to the otherwise traditional pipeline of factchecking in FEVER-like tasks. The authors treat BERT as a knowledge base and use its masked language modeling predictions to decide the factual correctness of the claim.…”

Section: Related Workmentioning

confidence: 99%

Automatic Fact-Checking with Document-level Annotations using BERT and Multiple Instance Learning

Sathe¹,

Park²

2021

Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER)

View full text Add to dashboard Cite

Automatic fact-checking is crucial for recognizing misinformation spreading on the internet. Most existing fact-checkers break down the process into several subtasks, one of which determines candidate evidence sentences that can potentially support or refute the claim to be verified; typically, evidence sentences with gold-standard labels are needed for this. In a more realistic setting, however, such sentence-level annotations are not available. In this paper, we tackle the natural language inference (NLI) subtask-given a document and a (sentence) claim, determine whether the document supports or refutes the claimonly using document-level annotations. Using fine-tuned BERT and multiple instance learning, we achieve 81.9% accuracy, significantly outperforming the existing results on the WikiFactCheck-English dataset.

show abstract

Language Models as Fact Checkers?

Cited by 32 publications

References 7 publications

Towards Few-shot Fact-Checking via Perplexity

Towards Few-shot Fact-Checking via Perplexity

Zero-shot Fact Verification by Claim Generation

Automatic Fact-Checking with Document-level Annotations using BERT and Multiple Instance Learning

Contact Info

Product

Resources

About