How Robust are Fact Checking Systems on Colloquial Claims?

Abstract

Knowledge is now starting to power neural dialogue agents. At the same time, the risk of misinformation and disinformation from dialogue agents also rises. Verifying the veracity of information from formal sources are widely studied in computational fact checking. In this work, we ask: How robust are fact checking systems on claims in colloquial style? We aim to open up new discussions in the intersection of fact verification and dialogue safety. In order to investigate how fact checking systems behave on colloquial claims, we transfer the styles of claims from FEVER (Thorne et al., 2018) into colloquialism. We find that existing fact checking systems that perform well on claims in formal style significantly degenerate on colloquial claims with the same semantics. Especially, we show that document retrieval is the weakest spot in the system even vulnerable to filler words, such as "yeah" and "you know". The document recall of WikiAPI retriever (Hanselowski et al., 2018) which is 90.0% on FEVER, drops to 72.2% on the colloquial claims. We compare the characteristics of colloquial claims to those of claims in formal style, and demonstrate the challenging issues in them.

Entity-based Claim Representation Improves Fact-Checking of Medical Content in Tweets

“…Along with Zuo et al (2022), they show that real-world medical claims in user-generated and news content are more complex and longer. In addition, Kim et al (2021) show that fact-checking systems do not transfer robustly to colloquial claims.…”

DialFact: A Benchmark for Fact-Checking in Dialogue

“…However, fact-checking in dialogue is still an unexplored area. Most recently, Kim et al (2021) explored fact-checking systems in the context of colloquial claims, which are conversation-style claims with personal opinions and informal comments. They curated colloquial claims by converting FEVER claims into colloquial style.…”

“…Colloquial (Kim et al, 2021) It contains colloquial claims converted from FEVER dataset claims into colloquial style (which include filler words, personal feelings, and informal comments). It has 410k colloquial claim-evidence pairs in the training set and is well aligned to our task because of its colloquial nature.…”

“…We next discuss the implementation details for the document retrieval methods. For WikiAPI method, Kim et al (2021) pointed out that Wiki-API method naively retrieves documents related to filler words such as "I", "Yes", "They" etc. frequently.…”

“…However, there is no data available for fact-checking in dialogue, and related work mainly focuses on improving factual consistency on knowledge-grounded response generation task (Honovich et al, 2021;Rashkin et al, 2021;Shuster et al, 2021). Most relevant to our work, Kim et al (2021) proposed the task of verification of colloquial claims which are stylistic modifications of the FEVER (Thorne et al, 2018) claims. However, their generated claims are not contextualized within dialogue.…”