GEMS: Generative Modeling for Evaluation of Summaries

Katragadda, Rahul

doi:10.1007/978-3-642-12116-6_61

Cited by 3 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast to the previous methods, the GEMS (Generative Modeling for Evaluation of Summaries) approach of Katragadda (2010) suggests the use of signatures terms to analyze how they are captured in peer summaries. Signature terms (also known as topic signatures) are word vectors related to a particular topic.…”

Section: Summary Contentmentioning

confidence: 99%

The challenging task of summary evaluation: an overview

Lloret

Plaza

Aker

2017

Lang Resources & Evaluation

View full text Add to dashboard Cite

Evaluation is crucial in the research and development of automatic summarization applications, in order to determine the appropriateness of a summary based on different criteria, such as the content it contains, and the way it is presented. To perform an adequate evaluation is of great relevance to ensure that automatic summaries can be useful for the context and/or application they are generated for. To this end, researchers must be aware of the evaluation metrics, approaches, and datasets that are available, in order to decide which of them would be the most suitable to use, or to be able to propose new ones, overcoming the possible limitations that existing methods may present. In this article, a critical and historical analysis of evaluation metrics, methods, and datasets for automatic summarization systems is presented, where the strengths and weaknesses of evaluation efforts are discussed and the major challenges to solve are identified. Therefore, a clear up-to-date overview of the evolution and progress of summarization evaluation is provided, giving the reader useful insights into the past, present and latest trends in the automatic evaluation of summaries.

show abstract

Section: Summary Contentmentioning

confidence: 99%

The challenging task of summary evaluation: an overview

Lloret

Plaza

Aker

2017

Lang Resources & Evaluation

View full text Add to dashboard Cite

show abstract

“…Whereas Basic Elements uses Minipar, 18 DEPEVAL(summ) is tested with different parsers, for instance the Charniak parser. 19 GEMS (Generative Modelling for Evaluation of Summaries) (Katragadda 2010) suggests the use of signature terms in order to analyse how they are captured in automatic summaries. The signature terms are calculated on the basis of part-of-speech tags, such as nouns or verbs; query terms and terms of reference summaries.…”

Section: Informativeness Evaluationmentioning

confidence: 99%

Text summarisation in progress: a literature review

2011

View full text Add to dashboard Cite

This paper contains a large literature review in the research field of Text Summarisation (TS) based on Human Language Technologies (HLT). TS helps users manage the vast amount of information available, by condensing documents' content and extracting the most relevant facts or topics included in them. The rapid development of emerging technologies poses new challenges to this research field, which still need to be solved. Therefore, it is essential to analyse its progress over the years, and provide an overview of the past, present and future directions, highlighting the main advances achieved and outlining remaining limitations. With this purpose, several important aspects are addressed within the scope of this survey. On the one hand, the paper aims at giving a general perspective on the state-ofthe-art, describing the main concepts, as well as different summarisation approaches, and relevant international forums. Furthermore, it is important to stress upon the fact that the birth of new requirements and scenarios has led to new types of summaries with specific purposes (e.g. sentiment-based summaries), and novel domains within which TS has proven to be also suitable for (e.g. blogs). In addition, TS is successfully combined with a number of intelligent systems based on HLT (e.g. information retrieval, question answering, and text classification). On the other hand, a deep study of the evaluation of summaries is also conducted in this paper, where the existing methodologies and systems are explained, as well as new research that has emerged concerning the automatic evaluation of summaries' quality. Finally, some thoughts about TS in general and its future will encourage the reader to think of novel approaches, applications and lines to conduct research in the next years. The analysis of these issues allows the reader to have a wide and useful background on the main important aspects of this research field.

show abstract

“…Researchers have questioned whether ROUGE is really able to capture the informativeness of summaries. Over the years, other automatic metrics have been proposed like Basic Elements (Hovy, Lin, Zhou, and Fukumoto, 2006), AutoSummENG (Giannakopoulos and Karkaletsis, 2009) and GEMS (Generative Modelling for Evaluation of Summaries) (Katragadda, 2010) to name a few, but none managed to demonstrate enough advantages to replace ROUGE as the standard evaluation metric. Lloret and Palomar (2011) discuss in detail these alternative evaluation metrics, whilst Owczarzak, Conroy, Dang, and Nenkova (2012) provide an assessment of evaluation metrics used in multi-document summarisation evaluations.…”

Section: Intrinsic Evaluation Methods In Text Summarisationmentioning

confidence: 99%

Automatic summarisation: 25 years On

Orăsan

2019

Nat. Lang. Eng.

View full text Add to dashboard Cite

Automatic text summarisation is a topic that has been receiving attention from the research community from the early days of computational linguistics, but it really took off around 25 years ago. This article presents the main developments from the last 25 years. It starts by defining what a summary is and how its definition changed over time as a result of the interest in processing new types of documents. The article continues with a brief history of the field and highlights the main challenges posed by the evaluation of summaries. The article finishes with some thoughts about the future of the field.

show abstract

GEMS: Generative Modeling for Evaluation of Summaries

Cited by 3 publications

References 9 publications

The challenging task of summary evaluation: an overview

The challenging task of summary evaluation: an overview

Text summarisation in progress: a literature review

Automatic summarisation: 25 years On

Contact Info

Product

Resources

About