Correlation between ROUGE and human evaluation of extractive meeting summaries

Liu, Feifan; Liu, Yang

doi:10.3115/1557690.1557747

Cited by 340 publications

(407 citation statements)

References 13 publications

Supporting

Mentioning

402

Contrasting

Unclassified

Order By: Relevance

“…Although the generated summaries were not good from a human's point of view, they obtained good results for some ROUGE metrics (for example, a recall score of 41% for ROUGE-1, which is acceptable in the state-of-the-art in this research field). In addition, the correlation between ROUGE and model summaries was shown to be lower than it was claimed, especially in some summarization types, such as in speech summarization (Liu and Liu, 2008b). Despite the need to have model summaries beforehand when using ROUGE, various researchers have shown that there is significant correlation between ROUGE scores and approaches based on human comparison of semantic content units (indeed this was necessary for ROUGE to win acceptance).…”

Section: Summary Contentmentioning

confidence: 97%

“…On the one hand, most content-oriented evaluation tools are based on content overlap, which presents a bias toward lexical similarity that may lead to unfair penalties when abstractive summaries are evaluated. However, it is interesting to mention that, in spite of this a priori disadvantageous situation, abstractive human summaries usually get significantly higher ROUGE scores (Liu and Liu, 2008a).…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

The challenging task of summary evaluation: an overview

Lloret

Plaza

Aker

2017

Lang Resources & Evaluation

View full text Add to dashboard Cite

Evaluation is crucial in the research and development of automatic summarization applications, in order to determine the appropriateness of a summary based on different criteria, such as the content it contains, and the way it is presented. To perform an adequate evaluation is of great relevance to ensure that automatic summaries can be useful for the context and/or application they are generated for. To this end, researchers must be aware of the evaluation metrics, approaches, and datasets that are available, in order to decide which of them would be the most suitable to use, or to be able to propose new ones, overcoming the possible limitations that existing methods may present. In this article, a critical and historical analysis of evaluation metrics, methods, and datasets for automatic summarization systems is presented, where the strengths and weaknesses of evaluation efforts are discussed and the major challenges to solve are identified. Therefore, a clear up-to-date overview of the evolution and progress of summarization evaluation is provided, giving the reader useful insights into the past, present and latest trends in the automatic evaluation of summaries.

show abstract

Section: Summary Contentmentioning

confidence: 97%

Section: Discussionmentioning

confidence: 99%

The challenging task of summary evaluation: an overview

Lloret

Plaza

Aker

2017

Lang Resources & Evaluation

View full text Add to dashboard Cite

show abstract

“…These TT set corresponds to the top x terms ranked based on the probability of the word given the topic (p(w|k)) from the topic model. We evaluated these summarisation approaches with the ROUGE-1 method (Lin, 2004), a widely used summarisation evaluation metric that correlates well with human evaluation (Liu and Liu, 2008). This method measures the overlap of words between the generated summary and a reference, in our case the GS generated from the NW dataset.…”

Section: Resultsmentioning

confidence: 99%

Automatic Labelling of Topic Models Learned from Twitter by Summarisation

Basave

2014

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

Latent topics derived by topic models such as Latent Dirichlet Allocation (LDA) are the result of hidden thematic structures which provide further insights into the data. The automatic labelling of such topics derived from social media poses however new challenges since topics may characterise novel events happening in the real world. Existing automatic topic labelling approaches which depend on external knowledge sources become less applicable here since relevant articles/concepts of the extracted topics may not exist in external sources. In this paper we propose to address the problem of automatic labelling of latent topics learned from Twitter as a summarisation problem. We introduce a framework which apply summarisation algorithms to generate topic labels. These algorithms are independent of external sources and only rely on the identification of dominant terms in documents related to the latent topic. We compare the efficiency of existing state of the art summarisation algorithms. Our results suggest that summarisation algorithms generate better topic labels which capture event-related context compared to the top-n terms returned by LDA.

show abstract

“…In particular, ROUGE-1, which works at the unigram level, was shown to significantly correlate with human evaluations. While it has been suggested than correlation may be weaker in the meeting domain (Liu and Liu, 2008), we stuck to ROUGE because 5 http://www-nlpir.nist.gov/projects/duc/ guidelines/2001.html of the lack of a clear substitute, and for consistency with the literature, as a very large majority of studies previously published in the domain use ROUGE.…”

Section: Discussionmentioning

confidence: 99%

Combining Graph Degeneracy and Submodularity for Unsupervised Extractive Summarization

Tixier¹,

Meladianos²,

Vazirgiannis³

2017

Proceedings of the Workshop on New Frontiers in Summarization

View full text Add to dashboard Cite

We present a fully unsupervised, extractive text summarization system that leverages a submodularity framework introduced by past research. The framework allows summaries to be generated in a greedy way while preserving near-optimal performance guarantees. Our main contribution is the novel coverage reward term of the objective function optimized by the greedy algorithm. This component builds on the graph-of-words representation of text and the k-core decomposition algorithm to assign meaningful scores to words. We evaluate our approach on the AMI and ICSI meeting speech corpora, and on the DUC2001 news corpus. We reach state-of-the-art performance on all datasets. Results indicate that our method is particularly well-suited to the meeting domain.

show abstract

Correlation between ROUGE and human evaluation of extractive meeting summaries

Cited by 340 publications

References 13 publications

The challenging task of summary evaluation: an overview

The challenging task of summary evaluation: an overview

Automatic Labelling of Topic Models Learned from Twitter by Summarisation

Combining Graph Degeneracy and Submodularity for Unsupervised Extractive Summarization

Contact Info

Product

Resources

About