Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics

Pagnoni, Artidoro; Balachandran, Vidhisha; Tsvetkov, Yulia

doi:10.48550/arxiv.2104.13346

Cited by 7 publications

(16 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the work of Pagnoni et al (2021), factual errors shaped as named entities and noun phrases account for the majority of all factual errors, while both are more in line with the task of cloze in our given scheme. As a result, we define named entities and noun phrases as factual factors in our approach.…”

Section: Factual Factorsmentioning

confidence: 56%

“…In contrast, Gabriel et al (2020); Pagnoni et al (2021) proposed the meta-evaluation of factual consistency with more refined criteria for the performance evaluation of factual consistency metrics.…”

Section: Related Workmentioning

confidence: 99%

“…The summaries used for evaluation are generated by a variety of different models on CNN/DM and XSum (Narayan et al, 2018), respectively. These datasets are XSumFaith (XSF) (Maynez et al, 2020), Sum-mEval (SE) (Fabbri et al, 2021a), FRANK (Pagnoni et al, 2021) and QAGS (Wang et al, 2020). We partition FRANK and QAGS into FRANK-CNN/DM (FCD), FRANK-XSum (FXS), QAGS-CNN/DM (QCD) and QAGS-XSum (QXS), resulting in a total of six test datasets.…”

Section: Factual Evaluation Benchmarkmentioning

confidence: 99%

“…Pagnoni et al (2021) included model summaries from CNN/DM and XSum datasets. They annotated each sentence of a summary with numerous error types based on their proposed criteria.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Just ClozE! A Fast and Simple Method for Evaluating the Factual Consistency in Abstractive Summarization

Li¹,

Li²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The issue of factual consistency in abstractive summarization has attracted much attention in recent years, and the evaluation of factual consistency between summary and document has become an important and urgent task. Most of the current evaluation metrics are adopted from the question answering (QA). However, the application of QA-based metrics is extremely time-consuming in practice, causing the iteration cycle of abstractive summarization research to be severely prolonged. In this paper, we propose a new method called ClozE to evaluate factual consistency by cloze model, instantiated based on masked language model(MLM), with strong interpretability and substantially higher speed. We demonstrate that ClozE can reduce the evaluation time by nearly 96% relative to QA-based metrics while retaining their interpretability and performance through experiments on six human-annotated datasets and a meta-evaluation benchmark GO FIGURE (Gabriel et al., 2020). We also implement experiments to further demonstrate more characteristics of ClozE in terms of performance and speed. In addition, we conduct an experimental analysis of the limitations of ClozE, which suggests future research directions. The code and models for ClozE will be released upon the paper acceptance.

show abstract

Section: Factual Factorsmentioning

confidence: 56%

Section: Related Workmentioning

confidence: 99%

Section: Factual Evaluation Benchmarkmentioning

confidence: 99%

“…Pagnoni et al (2021) included model summaries from CNN/DM and XSum datasets. They annotated each sentence of a summary with numerous error types based on their proposed criteria.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Just ClozE! A Fast and Simple Method for Evaluating the Factual Consistency in Abstractive Summarization

Li¹,

Li²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…A system capable of producing accurate summaries of the medical evidence on any given topic could dramatically improve the ability of caregivers to consult the whole of the evidence base to inform care. However, current neural summarization systems are prone to inserting inaccuracies into outputs (Kryscinski et al, 2020;Maynez et al, 2020;Pagnoni et al, 2021;Ladhak et al, 2021;Choubey et al, 2021). This has been shown specifically to be a problem in the context of medical literature summarization Otmakhova et al, 2022), where there is a heightened need for factual accuracy.…”

Section: Introductionmentioning

confidence: 99%

Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges

Ramprasad¹,

McInerney²,

Marshal³

et al. 2023

Preprint

View full text Add to dashboard Cite

We present TrialsSummarizer, a system that aims to automatically summarize evidence presented in the set of randomized controlled trials most relevant to a given query. Building on prior work , the system retrieves trial publications matching a query specifying a combination of condition, intervention(s), and outcome(s), and ranks these according to sample size and estimated study quality. The top-k such studies are passed through a neural multi-document summarization system, yielding a synopsis of these trials. We consider two architectures: A standard sequence-to-sequence model based on BART (Lewis et al., 2019), and a multi-headed architecture intended to provide greater transparency to end-users. Both models produce fluent and relevant summaries of evidence retrieved for queries, but their tendency to introduce unsupported statements render them inappropriate for use in this domain at present. The proposed architecture may help users verify outputs allowing users to trace generated tokens back to inputs. The demonstration video is available at: https://vimeo.com/735605060 The prototype, source code, and model weights are available at: https://sanjanaramprasad. github.io/trials-summarizer/.

show abstract

Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models

Chen,

Fu,

Yuan

et al. 2023

Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Large Language Models (LLMs) have gained widespread adoption in various natural language processing tasks, including question answering and dialogue systems. However, a major drawback of LLMs is the issue of hallucination, where they generate unfaithful or inconsistent content that deviates from the input source, leading to severe consequences. In this paper, we propose a robust discriminator named RelD to effectively detect hallucination in LLMs' generated answers. RelD is trained on the constructed RelQA, a bilingual question-answering dialogue dataset along with answers generated by LLMs and a comprehensive set of metrics. Our experimental results demonstrate that the proposed RelD successfully detects hallucination in the answers generated by diverse LLMs. Moreover, it performs well in distinguishing hallucination in LLMs' generated answers from both in-distribution and out-of-distribution datasets. Additionally, we also conduct a thorough analysis of the types of hallucinations that occur and present valuable insights. This research significantly contributes to the detection of reliable * Work done while this author was an intern at Microsoft Research.

show abstract

Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics

Cited by 7 publications

References 22 publications

Just ClozE! A Fast and Simple Method for Evaluating the Factual Consistency in Abstractive Summarization

Just ClozE! A Fast and Simple Method for Evaluating the Factual Consistency in Abstractive Summarization

Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges

Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models

Contact Info

Product

Resources

About