BERT for Evidence Retrieval and Claim Verification

Abstract

Motivated by the promising performance of pre-trained language models, we investigate BERT in an evidence retrieval and claim verification pipeline for the FEVER fact extraction and verification challenge. To this end, we propose to use two BERT models, one for retrieving potential evidence sentences supporting or rejecting claims, and another for verifying claims based on the predicted evidence sets. To train the BERT retrieval system, we use pointwise and pairwise loss functions, and examine the effect of hard negative mining. A second BERT model is trained to classify the samples as supported, refuted, and not enough information. Our system achieves a new state of the art recall of 87.1 for retrieving top five sentences out of the FEVER documents consisting of 50K Wikipedia pages, and scores second in the official leaderboard with the FEVER score of 69.7.

Fact or Fiction: Verifying Scientific Claims

“…We develop a baseline (referred to as VERISCI) that takes a claim c and corpus A as input, identifies evidence abstracts E(c), and predicts a label y(c, a) and rationale sentences S(c, a) for each a ∈ E(c). Following the "BERT-to-BERT" model presented in DeYoung et al (2020a); Soleimani et al (2019), VERISCI is a pipeline of three components: 1. ABSTRACTRETRIEVAL retrieves k abstracts with highest TF-IDF similarity to the claim.…”

Hierarchical Evidence Set Modeling for Automated Fact Extraction and Verification

“…Experiments are conducted to evaluate the performance of evidence retrieval, claim verification, and (Hanselowski et al, 2018b) 65.46 61.58 UCL MRG (Yoneda et al, 2018) 67.62 62.52 UNC NLP (Nie et al, 2019) 68.21 64.21 BERT Pair (Zhou et al, 2019) 69.75 65.18 BERT Concat (Zhou et al, 2019) 71.01 65.64 BERT (Base) (Soleimani et al, 2020) 70.67 68.50 GEAR (BERT Base) (Zhou et al, 2019) 71.60 67.10 KGAT (BERT Base) (Liu et al, 2020) 72 aggregation approaches. In addition, we conduct an ablation study.…”

“…Evidence sentence retrieval component in almost all previous works retrieves all the evidences through a single iteration (Yoneda et al, 2018;Hanselowski et al, 2018b;Nie et al, 2019;Chen et al, 2017;Soleimani et al, 2020;Liu et al, 2020). Stammbach and Neumann (2019) uses a multi-hop retrieval strategy through two iterations to retrieve evidence sentences that are conditioned on the retrieval of other evidence sentences.…”

“…In claim verification component, Nie et al (2019); Yoneda et al (2018); Hanselowski et al (2018b) use a modified ESIM model (Chen et al, 2017) for verification. Recent works (Soleimani et al, 2020;Zhou et al, 2019;Stammbach and Neumann, 2019) use BERT model (Devlin et al, 2019) for claim verification. Few other works (Zhou et al, 2019;Liu et al, 2020) use graph based models for fine-grained semantic reasoning.…”

“…All three models use a modified version of Enhanced Sequential Inference Model (Chen et al, 2017) Zhou et al (2019). Soleimani et al (2020) uses BERTbase and BERT-large for evidence retrieval and claim verification, respectively. They also experiment with both pairwise and point-wise ranking for evidence retrieval.…”