Summary Evaluation with and without References

Torres‐Moreno, Juan‐Manuel; Saggion, Horacio; Cunha, Iria da; SanJuan, Eric; Velázquez-Morales, Patricia

doi:10.17562/pb-42-2

Cited by 43 publications

(20 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Pyramid (Nenkova and Passonneau, 2004) and Responsiveness are the most popular such methods. The second group of methods is divided itself into two subsets: (1) methods that need human intervention like ROUGE (Lin, 2004a) and SERA (Cohan and Goharian, 2016), and (2) methods that do not need any human reference like SummTriver (Cabrera-Diego and Torres-Moreno, 2018) and FRESA (Torres-Moreno et al, 2010). The most popular automatic metric used by the community is ROUGE (Lin, 2004a).…”

Section: Related Workmentioning

confidence: 99%

“…Evaluation is usually done by humans, but manual evaluation is subjective, costly and time expensive (Lin and Hovy, 2002). Automatic evaluation methods (Lin, 2004a;Torres-Moreno et al, 2010;Zhao et al, 2019;Zhang et al, 2020) are an alternative to save time for users who extract the most relevant content from the web using Automatic Text Summarization systems (ATS). There exist two types of evaluation approaches: (1) manual evaluation methods like Pyramid (Nenkova and Passonneau, 2004) and Responsiveness, where human intervention is mandatory, and (2) automatic evaluation methods, where human intervention can be needed as a ground-truth reference (Lin, 2004a;Cohan and Goharian, 2016) or not (Torres-Moreno et al, 2010;Cabrera-Diego and Torres-Moreno, 2018).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

GeSERA: General-domain Summary Evaluation by Relevance Analysis

Espejel

Chalendar

Flores

et al. 2021

Proceedings of the Conference Recent Advances in Natural Language Processing - Deep Learning for Natural Language Processing Me

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

GeSERA: General-domain Summary Evaluation by Relevance Analysis

Espejel

Chalendar

Flores

et al. 2021

Proceedings of the Conference Recent Advances in Natural Language Processing - Deep Learning for Natural Language Processing Me

View full text Add to dashboard Cite

“…where 𝑃 is the probability distribution of words 𝑤 in text 𝑇 and 𝑄 is the probability distribution of words 𝑤 in summary 𝑆; 𝑁 is the number of words in text and summary 𝑁 = 𝑁 𝑇 + 𝑁 𝑆 , 𝐵 = 1.5|𝑉| where V is the size of the vocabulary of the documents, 𝐶 𝑤 𝑇 is the number of words in the text and 𝐶 𝑤 𝑇 is the number of words in the summary. For smoothing the summary's probabilities, we have used δ = 0.005 [53].…”

Section: Jensen-shannon Divergence (Js)mentioning

confidence: 99%

Ground Truth Spanish Automatic Extractive Text Summarization Bounds

Mendoza¹,

Ledeneva²,

Hernández³

et al. 2020

CyS

View full text Add to dashboard Cite

The textual information has accelerated growth in the most spoken languages by native Internet users, such as Chinese, Spanish, English, Arabic, Hindi, Portuguese, Bengali, Russian, among others. It is necessary to innovate the methods of Automatic Text Summarization (ATS) that can extract essential information without reading the entire text. The most competent methods are Extractive ATS (EATS) that extract essential parts of the document (sentences, phrases, or paragraphs) to compose a summary. During the last 60 years of research of EATS, the creation of standard corpus with human-generated summaries and evaluation methods which are highly correlated with human judgments help to increase the number of new state-of-the-art methods. However, these methods are mainly supported for the English language, leaving aside other equally important languages such as Spanish, which is the second most spoken language by natives and the third most used on the Internet. A standard corpus for Spanish EATS (SAETS) is created to evaluate the state-of-the-art methods and systems for the Spanish language. The main contribution consists of a proposal for configuration and evaluation of 5 state-ofthe-art methods, five systems and four heuristics using three evaluation methods (ROUGE, ROUGE-C, and Jensen-Shannon divergence). It is the first time that Jensen-Shannon divergence is used to evaluate AETS. In this paper the ground truth bounds for the Spanish language are presented, which are the heuristics baseline:first, baseline:random, topline and concordance. In addition, the ranking of 30 evaluation tests of the state-of-the-art methods and systems is calculated that forms a benchmark for SAETS.

show abstract

“…FRESA (Torres-Moreno et al, 2010) is a framework for the evaluation of summaries that relies on the Jensen-Shannon divergence between n-gram probabilities. It scores summaries directly against the source text without reference summaries.…”

Section: Related Workmentioning

confidence: 99%

Supporting content evaluation of student summaries by Idea Unit embedding

Gecchele¹,

Yamada

Tokunaga

et al. 2019

Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

This paper discusses the computer-assisted content evaluation of summaries. We propose a method to make a correspondence between the segments of the source text and its summary. As a unit of the segment, we adopt "Idea Unit (IU)" which is proposed in Applied Linguistics. Introducing IUs enables us to make a correspondence even for the sentences that contain multiple ideas. The IU correspondence is made based on the similarity between vector representations of IU. An evaluation experiment with two source texts and 20 summaries showed that the proposed method is more robust against rephrased expressions than the conventional ROUGEbased baselines. Also, the proposed method outperformed the baselines in recall. We implemented the proposed method in a GUI tool "Segment Matcher" that aids teachers to establish a link between corresponding IUs across the summary and source text.

show abstract

Summary Evaluation with and without References

Cited by 43 publications

References 20 publications

GeSERA: General-domain Summary Evaluation by Relevance Analysis

GeSERA: General-domain Summary Evaluation by Relevance Analysis

Ground Truth Spanish Automatic Extractive Text Summarization Bounds

Supporting content evaluation of student summaries by Idea Unit embedding

Contact Info

Product

Resources

About