Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning

Wu, Hanlu; Ma, Tengfei; Wu, Lingfei; Manyumwa, Tariro; Ji, Shouling

doi:10.18653/v1/2020.emnlp-main.294

Cited by 32 publications

(10 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…C) Reference-free Approach Rather than measuring the quality of candidate summary based on a ground truth summary, reference-free metrics for relevance measure the quality of candidate summary based on pseudo-reference summaries that are generated from source documents. Wu et al [117]'s proposed metric requires training samples of high-quality summaries for model supervision while other reference-free metrics can generate metric scores without the use of high-quality summaries as supervisory signals [14,35]. In section 3 of benchmark datasets, we see that the information covered by a reference summary depends on the data annotation approach as well as the intent of the original authors.…”

Section: A) Hard Lexical Overlapmentioning

confidence: 99%

An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics

Koh

Liu

et al. 2022

ACM Comput. Surv.

View full text Add to dashboard Cite

Long documents such as academic articles and business reports have been the standard format to detail out important issues and complicated subjects that require extra attention. An automatic summarization system that can effectively condense long documents into short and concise texts to encapsulate the most important information would thus be significant in aiding the reader’s comprehension. Recently, with the advent of neural architectures, significant research efforts have been made to advance automatic text summarization systems, and numerous studies on the challenges of extending these systems to the long document domain have emerged. In this survey, we provide a comprehensive overview of the research on long document summarization and a systematic evaluation across the three principal components of its research setting: benchmark datasets, summarization models, and evaluation metrics. For each component, we organize the literature within the context of long document summarization and conduct an empirical analysis to broaden the perspective on current research progress. The empirical analysis includes a study on the intrinsic characteristics of benchmark datasets, a multi-dimensional analysis of summarization models, and a review of the summarization evaluation metrics. Based on the overall findings, we conclude by proposing possible directions for future exploration in this rapidly growing field.

show abstract

Section: A) Hard Lexical Overlapmentioning

confidence: 99%

An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics

Koh

Liu

et al. 2022

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

“…More recently, MatchSum (Zhong et al 2020) formulates extractive summarization as a semantic text matching problem using contrastive learning. Wu et al (2020) measures the summary qualities without reference summaries by contrasting the document with the summaries using a ranking model. GSum (Dou et al 2021) takes different kinds of external guidance as additional input to the document and advances summarization performance significantly.…”

Section: Related Workmentioning

confidence: 99%

Sequence Level Contrastive Learning for Text Summarization

Zhang

et al. 2022

AAAI

View full text Add to dashboard Cite

Contrastive learning models have achieved great success in unsupervised visual representation learning, which maximize the similarities between feature representations of different views of the same image, while minimize the similarities between feature representations of views of different images. In text summarization, the output summary is a shorter form of the input document and they have similar meanings. In this paper, we propose a contrastive learning model for supervised abstractive text summarization, where we view a document, its gold summary and its model generated summaries as different views of the same mean representation and maximize the similarities between them during training. We improve over a strong sequence-to-sequence text generation model (i.e., BART) on three different summarization datasets. Human evaluation also shows that our model achieves better faithfulness ratings compared to its counterpart without contrastive objectives. We release our code at https://github.com/xssstory/SeqCo.

show abstract

“…(Shi et al, 2019) randomly replaces a sentence in the ground-truth summary with a random sentence to form the negative sample. Wu et al (2020) constructs negative samples on different aspects of summary qualities and propose a new summary evaluation method by contrastive learning. Zhong et al (2020) use a pre-trained extractive model to select several candidates as negative samples and take the groundtruth as the positive.…”

Section: Contrastive Learning In Summarizationmentioning

confidence: 99%

Contrastive Aligned Joint Learning for Multilingual Summarization

Wang¹,

Chen²,

Zhou³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Multilingual text summarization requires the ability to understand documents in multiple languages and generate summaries in the corresponding language, which poses more challenges on current summarization systems. However, this problem has been rarely studied due to the lack of large-scale supervised summarization data in multiple languages. In this paper, we first provide a large-scale multilingual summarization corpus MLGSum consisting of 1.1 million articles and summaries in 12 different languages. Based on it, we develop a unified summarization model to understand the document and generate summaries in different languages. We use the contrastive learning strategy to train our multilingual summarization system (CALMS), which consists of two training objectives, contrastive sentence ranking (CSR) and sentence aligned substitution (SAS). The two training objectives are designed to share salient information extractive ability and align sentencelevel representation across different languages. Experimental results indicate that CALMS achieves significant improvement over monolingual models in all languages. We further transfer CALMS to other languages and find that it will also benefit similar languages. Our code and dataset are available at https://github.com/brxx122/CALMS.

show abstract

Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning

Cited by 32 publications

References 22 publications

An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics

An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics

Sequence Level Contrastive Learning for Text Summarization

Contrastive Aligned Joint Learning for Multilingual Summarization

Contact Info

Product

Resources

About