DeSePtion: Dual Sequence Prediction and Adversarial Examples for Improved Fact-Checking

Hidey, Christopher; Chakrabarty, Tuhin; Alhindi, Tariq; Varia, Siddharth; Krstovski, Kriste; Diab, Mona; Muresan, Smaranda

doi:10.18653/v1/2020.acl-main.761

Cited by 38 publications

(28 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To select evidence sentences we follow the approach proposed by Hidey et al (2020). Given the true claims and the 5 evidence documents for each claim (Section 2.1) we use cosine similarity on SBERT sentence embeddings (Reimers and Gurevych, 2019) to extract the top 5 sentences most similar to the true claim.…”

Section: Evidence Sentence Selectionmentioning

confidence: 99%

COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic

Saakyan¹,

Chakrabarty²,

Muresan³

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

We introduce a FEVER-like dataset COVID-Fact of 4, 086 claims concerning the COVID-19 pandemic. The dataset contains claims, evidence for the claims, and contradictory claims refuted by the evidence. Unlike previous approaches, we automatically detect true claims and their source articles and then generate counter-claims using automatic methods rather than employing human annotators. Along with our constructed resource, we formally present the task of identifying relevant evidence for the claims and verifying whether the evidence refutes or supports a given claim. In addition to scientific claims, our data contains simplified general claims from media sources, making it better suited for detecting general misinformation regarding COVID-19. Our experiments indicate that COVID-Fact will provide a challenging testbed for the development of new systems and our approach will reduce the costs of building domainspecific datasets for detecting misinformation. Original ClaimClosed environments facilitate secondary transmission of coronavirus disease 2019 Counter-Claim Closed environments prevent secondary transmission of coronavirus disease 2019 Gold Document

show abstract

Section: Evidence Sentence Selectionmentioning

confidence: 99%

COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic

Saakyan¹,

Chakrabarty²,

Muresan³

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

show abstract

“…Another common setting for fact-checking is to assume a credible evidence source is given (e.g., Wikipedia), and to focus on the evidence retrieval and veracity verification steps only. FEVER (Thorne et al, 2018) and Tabfact are two large datasets for this setting, and there are many follow-up studies working on them (Yoneda et al, 2018a;Nie et al, 2019;Zhong et al, 2020;Herzig et al, 2020;Hidey et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

Towards Few-shot Fact-Checking via Perplexity

Lee¹,

Bang²,

Madotto³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Few-shot learning has drawn researchers' attention to overcome the problem of data scarcity. Recently, large pre-trained language models have shown great performance in fewshot learning for various downstream tasks, such as question answering and machine translation. Nevertheless, little exploration has been made to achieve few-shot learning for the fact-checking task. However, fact-checking is an important problem, especially when the amount of information online is growing exponentially every day. In this paper, we propose a new way of utilizing the powerful transfer learning ability of a language model via a perplexity score. The most notable strength of our methodology lies in its capability in fewshot learning. With only two training samples, our methodology can already outperform the Major Class baseline by more than an absolute 10% on the F1-Macro metric across multiple datasets. Through experiments, we empirically verify the plausibility of the rather surprising usage of the perplexity score in the context of fact-checking and highlight the strength of our few-shot methodology by comparing it to strong fine-tuning-based baseline models. Moreover, we construct and publicly release two new fact-checking datasets related to COVID-19.

show abstract

“…They follow up on this with the FEVER 2.0 task (Thorne et al, 2019b), where participants design adversarial attacks for existing FC systems. The first two winning systems (Niewinski et al, 2019;Hidey et al, 2020) produce claims requiring multi-hop reasoning, which has been shown to be challenging for fact checking models (Ostrowski et al, 2020). The other remaining system (Kim and Allan, 2019) generates adversarial attacks manually.…”

Section: Fact Checkingmentioning

confidence: 99%

Generating Label Cohesive and Well-Formed Adversarial Claims

Atanasova

Wright

Augenstein

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Adversarial attacks reveal important vulnerabilities and flaws of trained models. One potent type of attack are universal adversarial triggers, which are individual n-grams that, when appended to instances of a class under attack, can trick a model into predicting a target class. However, for inference tasks such as fact checking, these triggers often inadvertently invert the meaning of instances they are inserted in. In addition, such attacks produce semantically nonsensical inputs, as they simply concatenate triggers to existing samples. Here, we investigate how to generate adversarial attacks against fact checking systems that preserve the ground truth meaning and are semantically valid. We extend the HotFlip attack algorithm used for universal trigger generation by jointly minimizing the target class loss of a fact checking model and the entailment class loss of an auxiliary natural language inference model. We then train a conditional language model to generate semantically valid statements, which include the found universal triggers. We find that the generated attacks maintain the directionality and semantic validity of the claim better than previous work. * denotes equal contribution Dissociative disorders have been attributed to disruptions in memory caused by trauma or other forms of stress. SUPPORTS REFUTESDissociative identity disorder, or DID, may be the result of memory disruptions that have been induced by psychological trauma. CLAIMdon,already,more,during,home GPT-2 Claim GenerationDissociative disorders have been attributed to disrupted brain activity during trauma or other forms of stress.

show abstract

DeSePtion: Dual Sequence Prediction and Adversarial Examples for Improved Fact-Checking

Cited by 38 publications

References 41 publications

COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic

COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic

Towards Few-shot Fact-Checking via Perplexity

Generating Label Cohesive and Well-Formed Adversarial Claims

Contact Info

Product

Resources

About