Re-evaluating Evaluation in Text Summarization

Bhandari, Manik; Gour, Pranav; Ashfaq, Atabak; Liu, Pengfei; Neubig, Graham

doi:10.48550/arxiv.2010.07100

Cited by 5 publications

(10 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further, it is possible when the retrieved subset extracted by the content selection mechanism are not ordered in its original form, the incoherence of the subset cascade downwards to the inal summary output, causing a drop in semantic coherence. This inding also illustrates the importance of measuring model performance in a multi-dimensional way rather than relying entirely on ROUGE score that has found to have important limitations [3,61].…”

Section: 22mentioning

confidence: 84%

“…While both architectures were initially proposed and tested on short documents, they can be efectively adapted to summarize long documents after incorporating novel mechanisms. 3 Finding 1. Graph-based Extractive Models with Discourse Bias: Classical graph-based unsupervised extractive models have been found to sufer from picking similar sentences that results in a summary with redundant sentences [69].…”

Section: Supervised Hybridmentioning

confidence: 98%

“…The core idea of ROUGE score is to measure the lexical overlaps such as words and phrases between candidate summary and ground truth summary. While it is eicient, recent indings have shown that ROUGE score does not correlate well with how humans assess the quality of a candidate summary [3,11,44,61]. As a result, there is a signiicant amount of efort in improving the way we measure the quality of candidate summaries and performance of summarization architectures [62,80,81,126,130].…”

Section: Introductionmentioning

confidence: 99%

“…As a result, there is a signiicant amount of efort in improving the way we measure the quality of candidate summaries and performance of summarization architectures [62,80,81,126,130]. Unfortunately, these eforts have entirely been focusing on the short document domains and the progress in measuring the quality of long document summarization approach has been lacking [3,41,48,89,93].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics

Koh

Liu

et al. 2022

ACM Comput. Surv.

View full text Add to dashboard Cite

Long documents such as academic articles and business reports have been the standard format to detail out important issues and complicated subjects that require extra attention. An automatic summarization system that can effectively condense long documents into short and concise texts to encapsulate the most important information would thus be significant in aiding the reader’s comprehension. Recently, with the advent of neural architectures, significant research efforts have been made to advance automatic text summarization systems, and numerous studies on the challenges of extending these systems to the long document domain have emerged. In this survey, we provide a comprehensive overview of the research on long document summarization and a systematic evaluation across the three principal components of its research setting: benchmark datasets, summarization models, and evaluation metrics. For each component, we organize the literature within the context of long document summarization and conduct an empirical analysis to broaden the perspective on current research progress. The empirical analysis includes a study on the intrinsic characteristics of benchmark datasets, a multi-dimensional analysis of summarization models, and a review of the summarization evaluation metrics. Based on the overall findings, we conclude by proposing possible directions for future exploration in this rapidly growing field.

show abstract

Section: 22mentioning

confidence: 84%

Section: Supervised Hybridmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics

Koh

Liu

et al. 2022

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

“…We focus on five different tasks: summary evaluation, image description, dialogue and translation. For summary evaluation, we use TAC08 (Dang et al, 2008), TAC10, TAC11 (Owczarzak & Dang, 2011), RSUM (Bhandari et al, 2020) and SEVAL (Fabbri et al, 2021). For sentence-based image description we rely on FLICKR (Young et al, 2014) and for dialogue we use PersonaChat (PC) and TopicalChat (TC) (Mehri & Eskenazi, 2020).…”

Section: Datasets With Instance-level Informationmentioning

confidence: 99%

What are the best systems? New perspectives on NLP Benchmarking

Colombo¹,

Noiry²,

Irurozki³

et al. 2022

Preprint

View full text Add to dashboard Cite

In Machine Learning, a benchmark refers to an ensemble of datasets associated with one or multiple metrics together with a way to aggregate different systems performances. They are instrumental in (i) assessing the progress of new methods along different axes and (ii) selecting the best systems for practical use. This is particularly the case for NLP with the development of large pre-trained models (e.g. GPT, BERT) that are expected to generalize well on a variety of tasks. While the community mainly focused on developing new datasets and metrics, there has been little interest in the aggregation procedure, which is often reduced to a simple average over various performance measures. However, this procedure can be problematic when the metrics are on a different scale, which may lead to spurious conclusions. This paper proposes a new procedure to rank systems based on their performance across different tasks. Motivated by the social choice theory, the final system ordering is obtained through aggregating the rankings induced by each task and is theoretically grounded. We conduct extensive numerical experiments (on over 270k scores) to assess the soundness of our approach both on synthetic and real scores (e.g. GLUE, EXTREM, SE-VAL, TAC, FLICKR). In particular, we show that our method yields different conclusions on stateof-the-art systems than the mean-aggregation procedure while being both more reliable and robust.

show abstract

Accuracy is not enough: Evaluating Personalization in Summarizers

Vansh,

Rank,

Dasgupta

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Text summarization models are evaluated in terms of accuracy and quality using various measures such as ROUGE, BLEU, METEOR, BERTScore, PYRAMID, readability, and several recently proposed ones. The central objective of all accuracy measures is to evaluate the model's ability to capture saliency accurately. Since saliency is subjective w.r.t the readers' preferences, there cannot be a fit-all summary for a given document. This means that in many use cases, summarization models need to be personalized w.r.t user profiles. However, to our knowledge, there is no measure to evaluate the degree of personalization of a summarization model. In this paper, we first establish that existing accuracy measures cannot evaluate the degree of personalization of any summarization model and then propose a novel measure, called EGISES, for automatically computing the same. Using the PENS dataset released by Microsoft Research, we analyze the degree of personalization of ten different state-of-the-art summarization models (both extractive and abstractive), five of which are explicitly trained for personalized summarization, and the remaining are appropriated to exhibit personalization. We conclude by proposing a generalized accuracy measure, called P -Accuracy, for designing accuracy measures that should also take personalization into account and demonstrate the robustness and reliability of the measure through meta-evaluation.

show abstract

Re-evaluating Evaluation in Text Summarization

Cited by 5 publications

References 12 publications

An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics

An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics

What are the best systems? New perspectives on NLP Benchmarking

Accuracy is not enough: Evaluating Personalization in Summarizers

Contact Info

Product

Resources

About