Neural Text Summarization: A Critical Evaluation

Kryściński, Wojciech; Keskar, Nitish Shirish; McCann, Bryan; Xiong, Caiming; Socher, Richard

doi:10.18653/v1/d19-1051

Cited by 277 publications

(299 citation statements)

References 51 publications

Supporting

Mentioning

286

Contrasting

Unclassified

Order By: Relevance

“…Most automatic evaluation protocols, e.g., ROUGE and BERTScore [154], are not sufficient to evaluate the overall quality of generated summaries [70,91]. We still have to access some critical features, like factual correctness [70], fluency, and relevance [19], of generated summaries by human experts. Thus, a future research direction along this line is building better evaluation systems that go beyond current metrics to capture the most important features which agree with humans.…”

Section: Experiments On Newsroom and Bytecup Datasetsmentioning

confidence: 99%

Neural Abstractive Text Summarization with Sequence-to-Sequence Models

Shi

Keneshloo

Ramakrishnan

et al. 2021

ACM/IMS Trans. Data Sci.

144

View full text Add to dashboard Cite

In the past few years, neural abstractive text summarization with sequence-to-sequence (seq2seq) models have gained a lot of popularity. Many interesting techniques have been proposed to improve seq2seq models, making them capable of handling different challenges, such as saliency, fluency and human readability, and generate high-quality summaries. Generally speaking, most of these techniques differ in one of these three categories: network structure, parameter inference, and decoding/generation. There are also other concerns, such as efficiency and parallelism for training a model. In this article, we provide a comprehensive literature survey on different seq2seq models for abstractive text summarization from the viewpoint of network structures, training strategies, and summary generation algorithms. Several models were first proposed for language modeling and generation tasks, such as machine translation, and later applied to abstractive text summarization. Hence, we also provide a brief review of these models. As part of this survey, we also develop an open source library, namely, Neural Abstractive Text Summarizer (NATS) toolkit, for the abstractive text summarization. An extensive set of experiments have been conducted on the widely used CNN/Daily Mail dataset to examine the effectiveness of several different neural network components. Finally, we benchmark two models implemented in NATS on the two recently released datasets, namely, Newsroom and Bytecup.

show abstract

Section: Experiments On Newsroom and Bytecup Datasetsmentioning

confidence: 99%

Neural Abstractive Text Summarization with Sequence-to-Sequence Models

Shi

Keneshloo

Ramakrishnan

et al. 2021

ACM/IMS Trans. Data Sci.

144

View full text Add to dashboard Cite

show abstract

“…the input text (Cao et al, 2018). Automatic metrics used to evaluate text generation, such as ROUGE and BERTScore (Zhang et al, 2020), are not correlated with the factual consistency or faithfulness of the generated text (Falke et al, 2019;Kryściński et al, 2019). To address this, recent work has studied the use of textual entailment models to rank and filter non-factual generations (Falke et al, 2019;Maynez et al, 2020).…”

Section: Generaandonmentioning

confidence: 99%

Evaluating Factuality in Generation with Dependency-level Entailment

Goyal

Durrett

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Despite significant progress in text generation models, a serious limitation is their tendency to produce text that is factually inconsistent with information in the input. Recent work has studied whether textual entailment systems can be used to identify factual errors; however, these sentence-level entailment models are trained to solve a different problem than generation filtering and they do not localize which part of a generation is non-factual. In this paper, we propose a new formulation of entailment that decomposes it at the level of dependency arcs. Rather than focusing on aggregate decisions, we instead ask whether the semantic relationship manifested by individual dependency arcs in the generated output is supported by the input. Human judgments on this task are difficult to obtain; we therefore propose a method to automatically create data based on existing entailment or paraphrase corpora. Experiments show that our dependency arc entailment model trained on this data can identify factual inconsistencies in paraphrasing and summarization better than sentencelevel methods or those based on question generation, while additionally localizing the erroneous parts of the generation. 1

show abstract

“…The work also showed that ROUGE scores do not correlate with factual correctness, emphasizing that ROUGE based evaluation alone is not enough for summarization task. In addition, Kryscinski et al (2019a) pointed out that current evaluation protocols correlate weakly with human judgements and do not take factual correctness into account. Maynez et al (2020) conducted a large scale human evaluation on the generated summaries of various abstractive summarization systems and found substantial amounts of hallucinated content in those summaries.…”

Section: Related Workmentioning

confidence: 99%

“…Abstractive summarization has attracted increasing attention recently, thanks to the availability of large-scale datasets (Sandhaus, 2008;Hermann et al, 2015;Grusky et al, 2018;Narayan et al, 2018a) and advances on neural architectures (Sutskever et al, 2014;Bahdanau et al, 2015a;Vinyals et al, 2015;Vaswani et al, 2017). Although modern abstractive summarization systems generate relatively fluent summaries, recent work has called attention to the problem they have with factual inconsistency (Kryscinski et al, 2019a). That is, they produce summaries that contain hallucinated facts that are not supported by the source text.…”

Section: Introductionmentioning

confidence: 99%

Reducing Quantity Hallucinations in Abstractive Summarization

Zheng¹,

Cohen²,

Webber³

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

It is well-known that abstractive summaries are subject to hallucination-including material that is not supported by the original text. While summaries can be made hallucinationfree by limiting them to general phrases, such summaries would fail to be very informative. Alternatively, one can try to avoid hallucinations by verifying that any specific entities in the summary appear in the original text in a similar context. This is the approach taken by our system, HERMAN. The system learns to recognize and verify quantity entities (dates, numbers, sums of money, etc.) in a beamworth of abstractive summaries produced by state-of-the-art models, in order to up-rank those summaries whose quantity terms are supported by the original text. Experimental results demonstrate that the ROUGE scores of such up-ranked summaries have a higher Precision than summaries that have not been upranked, without a comparable loss in Recall, resulting in higher F 1 . Preliminary human evaluation of up-ranked vs. original summaries shows people's preference for the former.

show abstract

Neural Text Summarization: A Critical Evaluation

Cited by 277 publications

References 51 publications

Neural Abstractive Text Summarization with Sequence-to-Sequence Models

Neural Abstractive Text Summarization with Sequence-to-Sequence Models

Evaluating Factuality in Generation with Dependency-level Entailment

Reducing Quantity Hallucinations in Abstractive Summarization

Contact Info

Product

Resources

About