GO FIGURE: A Meta Evaluation of Factuality in Summarization

Gabriel, Saadia; Çelikyılmaz, Aslı; Jha, Rahul; Choi, Yejin; Gao, Jianfeng

doi:10.18653/v1/2021.findings-acl.42

Cited by 48 publications

(49 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the grounding of summary generation that was inherent to most traditional methods is yet to be achieved in neural summarization. The attention mechanism (Bahdanau et al, 2015), especially in pretrained encoder-decoder models (Lewis et al, 2020;Raffel et al, 2019;Zhang et al, 2020), plays a key role in aligning summary content to the input, yet undesired hallucinations are common in generated summaries (Maynez et al, 2020;Kryscinski et al, 2020;Gabriel et al, 2021).…”

mentioning

confidence: 99%

Planning with Learned Entity Prompts for Abstractive Summarization

Narayan

Zhao

Maynez

et al. 2021

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

We introduce a simple but flexible mechanism to learn an intermediate plan to ground the generation of abstractive summaries. Specifically, we prepend (or prompt) target summaries with entity chains—ordered sequences of entities mentioned in the summary. Transformer-based sequence-to-sequence models are then trained to generate the entity chain and then continue generating the summary conditioned on the entity chain and the input. We experimented with both pretraining and finetuning with this content planning objective. When evaluated on CNN/DailyMail, XSum, SAMSum, and BillSum, we demonstrate empirically that the grounded generation with the planning objective improves entity specificity and planning in summaries for all datasets, and achieves state-of-the-art performance on XSum and SAMSum in terms of rouge. Moreover, we demonstrate empirically that planning with entity chains provides a mechanism to control hallucinations in abstractive summaries. By prompting the decoder with a modified content plan that drops hallucinated entities, we outperform state-of-the-art approaches for faithfulness when evaluated automatically and by humans.

show abstract

mentioning

confidence: 99%

Planning with Learned Entity Prompts for Abstractive Summarization

Narayan

Zhao

Maynez

et al. 2021

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Basically, the revision strategy consists of three stages: (1) Phrase Trimming: remove phrases unsupported by source in the exemplar sentence; (2) Decontextualization: resolve co-reference and delete phrases dependent on context; (3) Syntax Modification: make the purified sentences flow smoothly. There are also some works [52,72] leveraging the model to generate data and instruct annotators to label whether these outputs contain hallucinations or not. While this approach is typically used to build diagnostic evaluation datasets, it has the potential to build faithful datasets.…”

Section: Hallucination Mitigation Methodsmentioning

confidence: 99%

“…The results show both QAGS and FEQA have substantially higher correlations with human judgments of faithfulness than the baseline metrics. In addition, Gabriel et al [52] further analyze the FEQA and find that the effectiveness of QA-based metrics depends on the question. They also provide a meta-evaluation framework that includes QA metrics.…”

Section: Hallucination Metrics In Abstractive Summarizationmentioning

confidence: 99%

Survey of Hallucination in Natural Language Generation

Ji¹,

Lee²,

Frieske³

et al. 2022

Preprint

View full text Add to dashboard Cite

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent natural language generation, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended texts, which degrades the system performance and fail to meet user expectations in many real-world scenarios. In order to address this issue, there have been studies in measuring and mitigating hallucinated texts. However there has not been a comprehensive review of the state-of-the-art in hallucination detection and mitigation.In this survey, we provide a broad overview of the research progress and challenges in the hallucination problem of NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; (2) an overview of task-specific research progress for hallucinations in a large set of downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, and machine translation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.

show abstract

“…Kryscinski et al (2019) note biases and failure modes of abstractive summarization models, while other work analyzes and collects annotations over the output of recent summarization models across multiple dimensions, including factual consistency Fabbri et al (2021); Bhandari et al (2020); Huang et al (2020). Lux et al (2020) propose a typology of errors found in summarization models, while Gabriel et al (2021) propose a framework for meta-evaluation of faithfulness consistency metrics. Laban et al (2021) propose to combine recent work in factual consistency evaluation for summarization through a single benchmark.…”

Section: Related Workmentioning

confidence: 99%

QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization

Fabbri¹,

Wu²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based metrics and question answering (QA)-based metrics. However, differing experimental setups presented in recent work lead to contrasting conclusions as to which paradigm performs best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFACTEVAL, that leads to a 15% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark. Our solution improves upon the best-performing entailment-based metric and achieves state-of-the-art performance on this benchmark. Furthermore, we find that QAbased and entailment-based metrics offer complementary signals and combine the two into a single, learned metric for further performance boost. Through qualitative and quantitative analyses, we point to question generation and answerability classification as two critical components for future work in QAbased metrics.

show abstract

GO FIGURE: A Meta Evaluation of Factuality in Summarization

Cited by 48 publications

References 31 publications

Planning with Learned Entity Prompts for Abstractive Summarization

Planning with Learned Entity Prompts for Abstractive Summarization

Survey of Hallucination in Natural Language Generation

QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization

Contact Info

Product

Resources

About