SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

Abstract

In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied to inconsistency detection. In this work, we revisit the use of NLI for inconsistency detection, finding that past work suffered from a mismatch in input granularity between NLI datasets (sentence-level), and inconsistency detection (document level). We provide a highly effective and light-weight method called SummaCConv that enables NLI models to be successfully used for this task by segmenting documents into sentence units and aggregating scores between pairs of sentences. We furthermore introduce a new benchmark called SummaC (Summary Consistency) which consists of six large inconsistency detection datasets. On this dataset, SummaCConv obtains state-of-the-art results with a balanced accuracy of 74.4%, a 5% improvement compared with prior work.

QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization

“…Metric comparisons, however, were often conducted on isolated datasets. Laban et al (2021) unify work in entailment-based metrics for factual consistency, showing the effect of granularity, base models, and other hyperparameter choices. This work also proposes a learned metric built on top of the output of an entailment model, with parameters fine-tuned on synthetic data.…”

“…For example, some work reports entailment-based metrics as performing best (Koto et al, 2020;Maynez et al, 2020), while other work argues for QA metrics (Durmus et al, 2020;Wang et al, 2020b;Scialom et al, 2021). Recently, Laban et al (2021) proposed a benchmark called SummaC to compare metrics across six factual consistency datasets for the task of binary factual consistency classification, whether a summary is entirely factually consistent or not. This work unifies prior work on entailment-based metrics by studying the effect of input granularity, pretrained entailment model, and other hyperparameter choices on downstream evaluation performance.…”

“…We study the effect of each of these components on metric performance. Based on our insights, we propose an optimized metric, which we call QAFACTEVAL , that outperforms the entailment-based metrics of Laban et al (2021).…”

“…Lux et al (2020) propose a typology of errors found in summarization models, while Gabriel et al (2021) propose a framework for meta-evaluation of faithfulness consistency metrics. Laban et al (2021) propose to combine recent work in factual consistency evaluation for summarization through a single benchmark. Our work directly makes use of this benchmark while emphasizing QA-based metrics.…”

“…SCZeroShot Laban et al (2021) works analogously to the above metrics with a base model trained on both MNLI and Vitamin-C data (Schuster et al, 2021), consisting of closely-related contrastive entailment examples.…”