Evidence Selection as a Token-Level Prediction Task

Abstract

In Automated Claim Verification, we retrieve evidence from a knowledge base to determine the veracity of a claim. Intuitively, the retrieval of the correct evidence plays a crucial role in this process. Often, evidence selection is tackled as a pairwise sentence classification task, i.e., we train a model to predict for each sentence individually whether it is evidence for a claim. In this work, we fine-tune document level transformers to extract all evidence from a Wikipedia document at once. We show that this approach performs better than a comparable model classifying sentences individually on all relevant evidence selection metrics in FEVER. Our complete pipeline building on this evidence selection procedure produces a new state-of-the-art result on FEVER, a popular claim verification benchmark.

ProoFVer: Natural Logic Theorem Proving for Fact Verification

“…Table 3 reports the fact verification results for ProoFVer and the baselines. Overall, ProoFVer-SB, our configuration using Stammbach's (2021) retriever, is the best performing model in our experiments. ProoFVer-SB, which outperforms Stammbach (2021) itself, is currently the highest scoring model in terms of label accuracy in the FEVER leaderboard.…”

“…We use CorefRoBERTA, their best-performing configuration. DominikS (Stammbach, 2021) focuses primarily on sentence-level evidence retrieval, scoring individual tokens from a given Wikipedia document, and then selecting the highest scoring sentences by averaging token scores. It uses a fine-tuned document level BigBird model (Zaheer et al, 2020) for this purpose.…”

“…ProoFVer's novelty lies in the proof generation in the third step. Hence, for better comparability, we follow two popular, wellperforming retrieval approaches, and Stammbach (2021). Liu et al's (2020) sentence retriever, also used in Ye et al (2020), is a sentence level pairwise ranking model, whereas that of Stammbach ( 2021) is a document level token score aggregation model.…”

“…To evaluate the robustness of fact verification systems against the impact of superfluous information from the retriever, we propose a new metric, Stability Error Rate (SER), which measures the proportion of instances where superfluous information changes the decision of the model. ProoFVer achieves a SER of 5.73%, compared with 9.36% of Stammbach (2021), where a lower SER is preferred. ProoFVer's proofs as explanations, apart from being faithful, score high in their overlap with human rationales with a token overlap F1-Score of 93.28%, 5.67 percentage points more than attention-based highlights from Ye et al (2020).…”

MultiVerS: Improving scientific claim verification with weak supervision and full-document context

“…Long-document encodings for fact verification have been explored by Stammbach (2021), who use Big Bird (Zaheer et al, 2020) for full-document evidence extraction from FEVER. Domain adaptation for scientific text has been studied in a number of works, including Gururangan et al ( 2020); Beltagy et al (2019); Lee et al (2020); Gu et al (2021).…”