HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification

Jiang, Yichen; Bordia, Shikha; Zheng, Zedong; Dognin, Charles; Singh, Maneesh; Bansal, Mohit

doi:10.18653/v1/2020.findings-emnlp.309

Cited by 62 publications

(77 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After selecting salient words from the true claims for replacement, we need to provide only paraphrases that are opposite in meaning and consider the context in which these words occur. Language models have been used previously for infilling tasks (Donahue et al, 2020) and have also been used for automatic claim mutation in fact checking (Jiang et al, 2020). Inspired by these approaches, we use the Masked Language Model (MLM) RoBERTa (Liu et al, 2019) fine-tuned on CORD-19 (Wang et al, 2020) for infilling.…”

Section: Masked Language Model Infilling With Entailment-based Quality Controlmentioning

confidence: 99%

COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic

Saakyan¹,

Chakrabarty²,

Muresan³

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

We introduce a FEVER-like dataset COVID-Fact of 4, 086 claims concerning the COVID-19 pandemic. The dataset contains claims, evidence for the claims, and contradictory claims refuted by the evidence. Unlike previous approaches, we automatically detect true claims and their source articles and then generate counter-claims using automatic methods rather than employing human annotators. Along with our constructed resource, we formally present the task of identifying relevant evidence for the claims and verifying whether the evidence refutes or supports a given claim. In addition to scientific claims, our data contains simplified general claims from media sources, making it better suited for detecting general misinformation regarding COVID-19. Our experiments indicate that COVID-Fact will provide a challenging testbed for the development of new systems and our approach will reduce the costs of building domainspecific datasets for detecting misinformation. Original ClaimClosed environments facilitate secondary transmission of coronavirus disease 2019 Counter-Claim Closed environments prevent secondary transmission of coronavirus disease 2019 Gold Document

show abstract

Section: Masked Language Model Infilling With Entailment-based Quality Controlmentioning

confidence: 99%

COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic

Saakyan¹,

Chakrabarty²,

Muresan³

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…Fact Checking and Verification. The need for claim verification has led to annotated fact check-ing datasets (Thorne et al, 2018;Baly et al, 2018;Augenstein et al, 2019;Jiang et al, 2020;Wadden et al, 2020;. Recent works deploy adversarial attacks against fact checking systems ( Thorne et al, 2019a,b;Niewinski et al, 2019;Atanasova et al, 2020b) and attempt to improve the system through generation (Atanasova et al, 2020a;Goyal and Durrett, 2020;Fan et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

“…Augenstein et al (2019) collect claims on fact checking websites and release the MultiFC dataset. Jiang et al (2020) collect a dataset requiring many-hop evidence extraction from Wikipedia. Wadden et al (2020) collect a dataset of scientific claims to be verified.…”

Section: Introductionmentioning

confidence: 99%

“…2.1 Fact Checking Pipeline FEVER (Thorne et al, 2018) is a fact checking benchmark dataset based on Wikipedia. Its fact checking pipeline has become one of the standard followed by many (Hanselowski et al, 2018;Nie et al, 2019;Zhou et al, 2019;Jiang et al, 2020). The pipeline comprises three stages: document retrieval, evidence selection, and claim verification.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

How Robust are Fact Checking Systems on Colloquial Claims?

Kim¹,

Kim²,

Hong³

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Knowledge is now starting to power neural dialogue agents. At the same time, the risk of misinformation and disinformation from dialogue agents also rises. Verifying the veracity of information from formal sources are widely studied in computational fact checking. In this work, we ask: How robust are fact checking systems on claims in colloquial style? We aim to open up new discussions in the intersection of fact verification and dialogue safety. In order to investigate how fact checking systems behave on colloquial claims, we transfer the styles of claims from FEVER (Thorne et al., 2018) into colloquialism. We find that existing fact checking systems that perform well on claims in formal style significantly degenerate on colloquial claims with the same semantics. Especially, we show that document retrieval is the weakest spot in the system even vulnerable to filler words, such as "yeah" and "you know". The document recall of WikiAPI retriever (Hanselowski et al., 2018) which is 90.0% on FEVER, drops to 72.2% on the colloquial claims. We compare the characteristics of colloquial claims to those of claims in formal style, and demonstrate the challenging issues in them.

show abstract

“…Given only a single statement and a single sentence, this decision process is called recognizing textual entailment (Dagan et al, 2010, RTE) or natural language inference (Bowman et al, 2015;Williams et al, 2018, NLI). Given a single statement and a vast pool of possible evidence (e.g., all of Wikipedia), this problem is called verification (Thorne et al, 2018;Jiang et al, 2020). In stages 1 to 4, players write challenging claims either entailed or refuted by evidence from Wikipedia (Section 3.1).…”

Section: Introducing a Game Of Challenging Claimsmentioning

confidence: 99%

Fool Me Twice: Entailment from Wikipedia Gamification

Eisenschlos¹,

Dhingra²,

Bulian³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

We release FOOLMETWICE (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game. Gamification encourages adversarial examples, drastically lowering the number of examples that can be solved using "shortcuts" compared to other entailment datasets. Players are presented with two tasks. The first task asks the player to write a plausible claim based on the evidence from a Wikipedia page. The second one shows two plausible claims written by other players, one of which is false, and the goal is to identify it before the time runs out. Players "pay" to see clues retrieved from the evidence pool: the more evidence the player needs, the harder the claim. Game-play between motivated players leads to diverse strategies for crafting claims, such as temporal inference and diverting to unrelated evidence, and results in higher quality data for the entailment and evidence retrieval tasks. We open source the dataset and game code. 1 * Work completed while a Visiting Research Scientist at Google.

show abstract

HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification

Cited by 62 publications

References 30 publications

COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic

COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic

How Robust are Fact Checking Systems on Colloquial Claims?

Fool Me Twice: Entailment from Wikipedia Gamification

Contact Info

Product

Resources

About