Explainable Automated Fact-Checking: A Survey

Kotonya, Neema; Toni, Francesca

doi:10.48550/arxiv.2011.03870

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [6], the authors note that the availability of datasets for experimentation is the current bottleneck with automatic claim verification. The two datasets that we first considered were LIAR [14] and LIAR-PLUS [16].…”

Section: Exclaim Foundational Datasetmentioning

confidence: 99%

“…Professional fact-checkers and independent factchecking organizations exist, but their work cannot be scaled linearly with the amount of digital content growth. This led to an increase in interest in automated fact-checking, also known as claim verification [6]. By fusing deep learning techniques with complex networks and algorithms, claim verification systems have achieved respectable performance but have become black-boxes.…”

Section: Introductionmentioning

confidence: 99%

“…The developers of the model and the end-users which we refer to as the nonexperts do not fully know the internals or understand its decision-making processes. Furthermore, [6] points that there exists no system that offers explanations for sub-tasks in the claim verification pipeline.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

ExClaim: Explainable Neural Claim Verification Using Rationalization

Gurrapu,

Huang,

Batarseh

2023

Preprint

View full text Add to dashboard Cite

With the advent of deep learning, text generation language models have improved dramatically, with text at a similar level as human-written text. This can lead to rampant misinformation because content can now be created cheaply and distributed quickly. Automated claim verification methods exist to validate claims, but they lack foundational data and often use mainstream news as evidence sources that are strongly biased towards a specific agenda. Current claim verification methods use deep neural network models and complex algorithms for a high classification accuracy but it is at the expense of model explainability. The models are black-boxes and their decisionmaking process and the steps it took to arrive at a final prediction are obfuscated from the user. We introduce a novel claim verification approach, namely: ExClaim, that attempts to provide an explainable claim verification system with foundational evidence. Inspired by the legal system, ExClaim leverages rationalization to provide a verdict for the claim and justifies the verdict through a natural language explanation (rationale) to describe the model's decision-making process. ExClaim treats the verdict classification task as a question-answer problem and achieves a performance of 0.93 F1 score. It provides subtasks explanations to also justify the intermediate outcomes. Statistical and Explainable AI (XAI) evaluations are conducted to ensure valid and trustworthy outcomes. Ensuring claim verification systems are assured, rational, and explainable is an essential step toward improving Human-AI trust and the accessibility of black-box systems.

show abstract

Section: Exclaim Foundational Datasetmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ExClaim: Explainable Neural Claim Verification Using Rationalization

Gurrapu,

Huang,

Batarseh

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…[2021] overviews the emerging tasks of claim detection and claim validation, and finally presents a comprehensive and up-to-date survey that highlights research challenges. Publicly available datasets have been gradually improving in terms of scale , Sathe et al, 2020, Aly et al, 2021, enriched features [Augenstein et al, 2019, Ostrowski et al, 2020, Kotonya and Toni, 2020b, on-demand domains [Wadden et al, 2020, Diggelmann et al, 2021, Saakyan et al, 2021, and novel perspectives , Schuster et al, 2021. Recently proposed systems address various challenges, e.g.…”

Section: Related Workmentioning

confidence: 99%

Aggregating Pairwise Semantic Differences for Few-Shot Claim Veracity Classification

Zeng¹,

Zubiaga²

2022

Preprint

View full text Add to dashboard Cite

As part of an automated fact-checking pipeline, the claim veracity classification task consists in determining if a claim is supported by an associated piece of evidence. The complexity of gathering labelled claim-evidence pairs leads to a scarcity of datasets, particularly when dealing with new domains. In this paper, we introduce SEED, a novel vector-based method to few-shot claim veracity classification that aggregates pairwise semantic differences for claim-evidence pairs. We build on the hypothesis that we can simulate class representative vectors that capture average semantic differences for claim-evidence pairs in a class, which can then be used for classification of new instances. We compare the performance of our method with competitive baselines including finetuned BERT/RoBERTa models, as well as the state-of-the-art few-shot veracity classification method that leverages language model perplexity. Experiments conducted on the FEVER and SCIFACT datasets show consistent improvements over competitive baselines in few-shot settings. Our code is available.

show abstract

Scientific evidence and specific context: leveraging large language models for health fact-checking

Ni,

Qian,

Chen

et al. 2024

OIR

View full text Add to dashboard Cite

PurposeThis study aims to evaluate the performance of LLMs with various prompt engineering strategies in the context of health fact-checking.Design/methodology/approachInspired by Dual Process Theory, we introduce two kinds of prompts: Conclusion-first (System 1) and Explanation-first (System 2), and their respective retrieval-augmented variations. We evaluate the performance of these prompts across accuracy, argument elements, common errors and cost-effectiveness. Our study, conducted on two public health fact-checking datasets, categorized 10,212 claims as knowledge, anecdotes and news. To further analyze the reasoning process of LLM, we delve into the argument elements of health fact-checking generated by different prompts, revealing their tendencies in using evidence and contextual qualifiers. We conducted content analysis to identify and compare the common errors across various prompts.FindingsResults indicate that the Conclusion-first prompt performs well in knowledge (89.70%,66.09%), anecdote (79.49%,79.99%) and news (85.61%,85.95%) claims even without retrieval augmentation, proving to be cost-effective. In contrast, the Explanation-first prompt often classifies claims as unknown. However, it significantly boosts accuracy for news claims (87.53%,88.60%) and anecdote claims (87.28%,90.62%) with retrieval augmentation. The Explanation-first prompt is more focused on context specificity and user intent understanding during health fact-checking, showing high potential with retrieval augmentation. Additionally, retrieval-augmented LLMs concentrate more on evidence and context, highlighting the importance of the relevance and safety of retrieved content.Originality/valueThis study offers insights into how a balanced integration could enhance the overall performance of LLMs in critical applications, paving the way for future research on optimizing LLMs for complex cognitive tasks.Peer reviewThe peer review history for this article is available at: https://publons.com/publon/10.1108/OIR-02-2024-0111

show abstract

Explainable Automated Fact-Checking: A Survey

Cited by 3 publications

References 0 publications

ExClaim: Explainable Neural Claim Verification Using Rationalization

ExClaim: Explainable Neural Claim Verification Using Rationalization

Aggregating Pairwise Semantic Differences for Few-Shot Claim Veracity Classification

Scientific evidence and specific context: leveraging large language models for health fact-checking

Contact Info

Product

Resources

About