Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics

Bhandari, Manik; Gour, Pranav; Ashfaq, Atabak; Liu, Pengfei

doi:10.48550/arxiv.2011.04096

Cited by 1 publication

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further, ROUGE does not work when reference summaries are not available. During the last two years, there has been a spurt in research related to metrics for summary quality (Peyrard, 2019b;Bhandari et al, 2020a;Huang et al, 2020;Vasilyev & Bohannon, 2020;Fabbri et al, 2020;Bhandari et al, 2020b). Most of these works have argued against the ROUGE metric because it fails to robustly match paraphrases resulting in misleading scores, which do not correlate well with human judgements (Zhang et al, 2019;Huang et al, 2020).…”

Section: Evaluation Metricsmentioning

confidence: 99%

See 1 more Smart Citation

Investigating Entropy for Extractive Document Summarization

Khurana,

Bhatnagar

2021

Preprint

View full text Add to dashboard Cite

Automatic text summarization aims to cut down readers' time and cognitive effort by reducing the content of a text document without compromising on its essence. Ergo, informativeness is the prime attribute of document summary generated by an algorithm, and selecting sentences that capture the essence of a document is the primary goal of extractive document summarization.In this paper, we employ Shannon's entropy to capture informativeness of sentences. We employ Non-negative Matrix Factorization (NMF) to reveal probability distributions for computing entropy of terms, topics, and sentences in latent space. We present an information theoretic interpretation of the computed entropy, which is the bedrock of the proposed E-Summ algorithm, an unsupervised method for extractive document summarization. The algorithm systematically applies information theoretic principle for selecting informative sentences from important topics in the document. The proposed algorithm is generic and fast, and hence amenable to use for summarization of documents in real time. Furthermore, it is domain-, collection-independent and agnostic to the language of the document. Benefiting from strictly positive NMF factor matrices, E-Summ algorithm is transparent and explainable too.We use standard ROUGE toolkit for performance evaluation of the proposed method on four well known public data-sets. We also perform quantitative as- *

show abstract

Section: Evaluation Metricsmentioning

confidence: 99%

“…Further, recent debate and consequent surge in study of evaluation metrics for automatic summaries is a clear and strong testimony to the considerable complexity of the task (Peyrard, 2019a,b;Ermakova et al, 2019;Bhandari et al, 2020a;Vasilyev & Bohannon, 2020;Fabbri et al, 2020;Huang et al, 2020;Bhandari et al, 2020b).…”

Section: Introductionmentioning

confidence: 99%

Investigating Entropy for Extractive Document Summarization

Khurana,

Bhatnagar

2021

Preprint

View full text Add to dashboard Cite

show abstract

Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics

Cited by 1 publication

References 9 publications

Investigating Entropy for Extractive Document Summarization

Investigating Entropy for Extractive Document Summarization

Contact Info

Product

Resources

About