Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims

Abstract

We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. In its starting year, the lab featured two tasks. Task 1 asked to predict which (potential) claims in a political debate should be prioritized for factchecking; in particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact checking. Task 2 asked to assess whether a given check-worthy claim made by a politician in the context of a debate/speech is factually true, half-true, or false. We offered both tasks in English and in Arabic. In terms of data, for both tasks, we focused on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign (we also provided translations in Arabic), and we relied on comments and factuality judgments from factcheck.org and snopes.com, which we further refined manually. A total of 30 teams registered to participate in the lab, and 9 of them actually submitted runs. The evaluation results show that the most successful approaches used various neural networks (esp. for Task 1) and evidence retrieval from the Web (esp. for Task 2). We release all datasets, the evaluation scripts, and the submissions by the participants, which should enable further research in both check-worthiness estimation and in automatic claim verification.

Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias

“…Kriplean et al [6] analyse how volunteer crowdsourcing can be used for fact-checking by simulating the democratic process. The Fact-checking Lab at CLEF [3,9] looks at this problem by defining the task of ranking sentences according to their need to be fact-checked. Maddalena et al [7] focus on the ability of the crowd to assess news quality along eight different quality dimensions.…”

Neural Check-Worthiness Ranking with Weak Supervision: Finding Sentences for Fact-Checking

“…CLEF 2018 competition on check-worthiness detection [17] but using even more data). Our model does not make use of specialized hand-crafted features as most related work [7,8,10,19], but instead adapts the model and its representation to the domain by being trained in an end-to-end fashion.…”

“…The work by Gencheva et al has been extended to both English and Arabic [10]. In the recent CLEF 2018 competition on check-worthiness detection [17], Zou et al [26] came first by using a large set of features (similarly to the above mentioned work), and doing feature selection with both a χ 2 -test and a linear SVM using a L1 regularizer.…”

CheckThat! at CLEF 2020: Enabling the Automatic Identification and Verification of Claims in Social Media

“…Our tasks cover all four steps. (Box 1 maps to task 1 whereas boxes 3-4 map to task 2 of the 2018 and 2019 editions [10,29]. )…”

“…at CLEF 2020 is the third edition of the lab. The 2018 edition [29] of CheckThat! focused on the identification and verification of claims in political debates.…”