Claim Check-Worthiness Detection as Positive Unlabelled Learning

Abstract

As the first step of automatic fact checking, claim check-worthiness detection is a critical component of fact checking systems. There are multiple lines of research which study this problem: check-worthiness ranking from political speeches and debates, rumour detection on Twitter, and citation needed detection from Wikipedia. To date, there has been no structured comparison of these various tasks to understand their relatedness, and no investigation into whether or not a unified approach to all of them is achievable. In this work, we illuminate a central challenge in claim check-worthiness detection underlying all of these tasks, being that they hinge upon detecting both how factual a sentence is, as well as how likely a sentence is to be believed without verification. As such, annotators only mark those instances they judge to be clear-cut check-worthy. Our best performing method is a unified approach which automatically corrects for this using a variant of positive unlabelled learning that finds instances which were incorrectly labelled as not check-worthy. In applying this, we outperform the state of the art in two of the three tasks studied for claim check-worthiness detection in English.

Generating Label Cohesive and Well-Formed Adversarial Claims

“…Fact checking systems consist of components to identify check-worthy claims (Atanasova et al, 2018;Wright and Augenstein, 2020), retrieve and rank evidence documents (Yin and Roth, 2018;Allein et al, 2020), determine the relationship between claims and evidence documents (Bowman et al, 2015;Augenstein et al, 2016;Baly et al, 2018), and finally predict the claims' veracity (Thorne et al, 2018;Augenstein et al, 2019). As this is a relatively involved task, models easily overfit to shallow textual patterns, necessitating the need for adversarial examples to evaluate the limits of their performance.…”

Determining the Credibility of Science Communication

“…We find that the best performance can be achieved by a Longformer-based model (Beltagy et al, 2020), which encodes entire paragraphs in papers and jointly predicts cite-worthiness labels for each of the sentences contained in the paragraph. Additional gains in recall can be achieved by using positive unlabelled learning, as documented in Wright and Augenstein (2020a) for the related task Exaggerated Claims Press Release: Players of the game rock paper scissors subconsciously copy each other's hand shapes, significantly increasing the chance of the game ending in a draw, according to new research.…”

“…CITEWORTH consists of 1.2M sentences, balanced across 10 diverse scientific fields. While others have studied this task for few and/or narrow domains (Sugiyama et al, 2010;Färber et al, 2018), and have also studied very related tasks, such as claim check-worthiness detection (Wright and Augenstein, 2020a) or citation recommendation (Jürgens et al, 2018), this is the largest and most diverse dataset for this task to date.…”

CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding

“…SciBERT + PU Learning We experiment with SciBERT trained using positive-unlabelled (PU) learning (Elkan and Noto, 2008) which has been shown to significantly improve performance on citation needed detection in Wikipedia and rumour detection on Twitter (Wright and Augenstein, 2020a). The intuition behind PU learning is to assume that cite-worthy data is labelled and noncite-worthy data is unlabelled, containing some cite-worthy examples.…”