The CLEF-2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News

Abstract

We describe the fourth edition of the CheckThat! Lab, part of the 2021 Cross-Language Evaluation Forum (CLEF). The lab evaluates technology supporting various tasks related to factuality, and it is offered in Arabic, Bulgarian, English, and Spanish. Task 1 asks to predict which tweets in a Twitter stream are worth fact-checking (focusing on COVID-19). Task 2 asks to determine whether a claim in a tweet can be verified using a set of previously fact-checked claims. Task 3 asks to predict the veracity of a target news article and its topical domain. The evaluation is carried out using mean average precision or precision at rank k for the ranking tasks, and F1 for the classification tasks.

Automated Fact-Checking for Assisting Human Fact-Checkers

“…This has encouraged the development of AI solutions, e.g., as part of shared tasks such as the CLEF CheckThat! lab 2018-2021 Elsayed et al, 2019;Barrón-Cedeño et al, 2020;Nakov et al, 2021a], and inside dedicated fact-checking organizations such as Full Fact. 12 The problem is widely tackled as a ranking one, where the system has to produce a check-worthiness scores.…”

“…The task was also featured in the CLEF CheckThat! Lab [Barrón-Cedeño et al, 2020;Nakov et al, 2021a]. Vo and Lee [2020] explored a multi-modal setup, where tweets with claims about images were matched against the Fauxtography section of Snopes.…”

“…A large body of research focused on developing automatic systems for fact-checking [Li et al, 2016;Shu et al, 2017;Lazer et al, 2018;Vosoughi et al, 2018;Vo and Lee, 2018]. This includes datasets [Hassan et al, 2015;Augenstein et al, 2019], and evaluation campaigns [Thorne and Vlachos, 2018;Nakov et al, 2021a]. However, there are credibility issues with automated systems [Arnold, 2020], and thus a reasonable solution is to build tools to facilitate human factcheckers.…”

Automated fact‐checking: A survey

“…Going further, one can also distinguish between check-worthy versus non-check-worthy claims (Nakov et al, 2021), as cases that being claims are worthy or not fact-checking. For example, one could argue that 'the government invested more than 10 billion last year in education' is a claim that is worthy of fact-checking, whereas a claim such as 'my friend had a coffee this morning for breakfast' may not be worthy of fact-checking.…”

“…A similar pipeline was proposed by CLEF CheckThat! (Nakov et al, 2021), which in its 2021 edition included three subtasks: firstly, perform claim detection to detect claims that are check-worthy; secondly, determine whether a claim has been previously fact-checked; and thirdly, perform claim validation to determine the factuality of the detected claims. While some pipelines include claim detection, some are only designed to tackle claim validation, for example, FEVER (Thorne et al, 2019; and SCIVER (Wadden et al, 2020), 1 assuming check-worthy claims are already at hand.…”