Overview of the CLEF-2024 CheckThat! Lab: Check-Worthiness, Subjectivity, Persuasion, Roles, Authorities, and Adversarial Robustness

Barrón-Cedeño, Alberto; Alam, Firoj; Struß, Julia Maria; Nakov, Preslav; Chakraborty, Tanmoy; Elsayed, Tamer; Przybyła, Piotr; Caselli, Tommaso; Da San Martino, Giovanni; Haouari, Fatima; Hasanain, Maram; Li, Chengkai; Piskorski, Jakub; Ruggeri, Federico; Song, Xingyi; Suwaileh, Reem

doi:10.1007/978-3-031-71908-0_2

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2024

Publication Types

Select...

Book1

Article1

Relationship

Self Cite0

Independent2

Authors

Journals

Cited by 2 publications

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

What Happened in CLEF$$\ldots $$ For Another While?

Ferro

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

What Happened in CLEF$$\ldots $$ For Another While?

Ferro

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Verifying the robustness of automatic credibility assessment

Przybyła,

Shvets,

Saggion

2024

Nat. lang. processing

View full text Add to dashboard Cite

Text classification methods have been widely investigated as a way to detect content of low credibility: fake news, social media bots, propaganda, etc. Quite accurate models (likely based on deep neural networks) help in moderating public electronic platforms and often cause content creators to face rejection of their submissions or removal of already published texts. Having the incentive to evade further detection, content creators try to come up with a slightly modified version of the text (known as an attack with an adversarial example) that exploit the weaknesses of classifiers and result in a different output. Here we systematically test the robustness of common text classifiers against available attacking techniques and discover that, indeed, meaning-preserving changes in input text can mislead the models. The approaches we test focus on finding vulnerable spans in text and replacing individual characters or words, taking into account the similarity between the original and replacement content. We also introduce BODEGA: a benchmark for testing both victim models and attack methods on four misinformation detection tasks in an evaluation framework designed to simulate real use cases of content moderation. The attacked tasks include (1) fact checking and detection of (2) hyperpartisan news, (3) propaganda, and (4) rumours. Our experimental results show that modern large language models are often more vulnerable to attacks than previous, smaller solutions, e.g. attacks on GEMMA being up to 27% more successful than those on BERT. Finally, we manually analyse a subset adversarial examples and check what kinds of modifications are used in successful attacks.

show abstract

Overview of the CLEF-2024 CheckThat! Lab: Check-Worthiness, Subjectivity, Persuasion, Roles, Authorities, and Adversarial Robustness

Cited by 2 publications

References 37 publications

What Happened in CLEF$$\ldots $$ For Another While?

What Happened in CLEF$$\ldots $$ For Another While?

Verifying the robustness of automatic credibility assessment

Contact Info

Product

Resources

About