SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization

Gao, Yang; Zhao, Wei; Eger, Steffen

doi:10.18653/v1/2020.acl-main.124

Cited by 75 publications

(52 citation statements)

References 32 publications

(34 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also test prior predictions from a state-ofthe-art summary scoring method, SUPERT (Gao et al, 2020), which uses a variant of BERT that has been fine-tuned on news articles to obtain 1024-dimensional contextualized embeddings of a summary. To score a summary, SUPERT extracts a pseudo-reference summary from the source documents, then compares its embedding with that of the test summary.…”

Section: Methodsmentioning

confidence: 99%

Interactive Text Ranking with Bayesian Optimization: A Case Study on Community QA and Summarization

Simpson

Gao

Gurevych

2020

Transactions of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

For many NLP applications, such as question answering and summarization, the goal is to select the best solution from a large space of candidates to meet a particular user’s needs. To address the lack of user or task-specific training data, we propose an interactive text ranking approach that actively selects pairs of candidates, from which the user selects the best. Unlike previous strategies, which attempt to learn a ranking across the whole candidate space, our method uses Bayesian optimization to focus the user’s labeling effort on high quality candidates and integrate prior knowledge to cope better with small data scenarios. We apply our method to community question answering (cQA) and extractive multidocument summarization, finding that it significantly outperforms existing interactive approaches. We also show that the ranking function learned by our method is an effective reward function for reinforcement learning, which improves the state of the art for interactive summarization.

show abstract

Section: Methodsmentioning

confidence: 99%

Interactive Text Ranking with Bayesian Optimization: A Case Study on Community QA and Summarization

Simpson

Gao

Gurevych

2020

Transactions of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Some work discussed how to evaluate the quality of generated text in the reference-free setting (Louis and Nenkova, 2013;Peyrard et al, 2017;Peyrard and Gurevych, 2018;Shimanaka et al, 2018;Xenouleas et al, 2019;Sun and Nenkova, 2019;Böhm et al, 2019;Chen et al, 2018;Gao et al, 2020). Louis and Nenkova (2013), Peyrard et al (2017) and Peyrard and Gurevych (2018) leveraged regression models to fit human judgement.…”

Section: Reference-free Metricsmentioning

confidence: 99%

“…SUPERT generates pseudo references and evaluates the quality of the test summaries by calculating word mover's distance between the pseudo reference summaries and the test summaries (Gao et al, 2020). It is similar to MoverScore (Zhao et al, 2019) which uses the human-authored references instead of pseudo references.…”

Section: Reference-free Metricsmentioning

confidence: 99%

Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning

Wu¹,

Ma²,

Wu³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Evaluation of a document summarization system has been a critical factor to impact the success of the summarization task. Previous approaches, such as ROUGE, mainly consider the informativeness of the assessed summary and require human-generated references for each test summary. In this work, we propose to evaluate the summary qualities without reference summaries by unsupervised contrastive learning. Specifically, we design a new metric which covers both linguistic qualities and semantic informativeness based on BERT. To learn the metric, for each summary, we construct different types of negative samples with respect to different aspects of the summary qualities, and train our model with a ranking loss. Experiments on Newsroom and CNN/Daily Mail demonstrate that our new evaluation method outperforms other metrics even without reference summaries. Furthermore, we show that our method is general and transferable across datasets.

show abstract

“…The second approach extends the (single) reference into multiple ones, by automatically generating paraphrases of the reference (a.k.a pseudoreferences) (Albrecht and Hwa, 2008;Yoshimura et al, 2019;Kauchak and Barzilay, 2006;Edunov et al, 2018;Gao et al, 2020). Our method ( §3.3) follows this paradigm.…”

Section: Generation Evaluationmentioning

confidence: 99%

Semantically Driven Sentence Fusion: Modeling and Evaluation

Ben-David

Keller

Malmi

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Sentence fusion is the task of joining related sentences into coherent text. Current training and evaluation schemes for this task are based on single reference ground-truths and do not account for valid fusion variants. We show that this hinders models from robustly capturing the semantic relationship between input sentences. To alleviate this, we present an approach in which ground-truth solutions are automatically expanded into multiple references via curated equivalence classes of connective phrases. We apply this method to a large-scale dataset and use the augmented dataset for both model training and evaluation. To improve the learning of semantic representation using multiple references, we enrich the model with auxiliary discourse classification tasks under a multi-tasking framework. Our experiments highlight the improvements of our approach over state-of-the-art models. 1

show abstract

SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization

Cited by 75 publications

References 32 publications

Interactive Text Ranking with Bayesian Optimization: A Case Study on Community QA and Summarization

Interactive Text Ranking with Bayesian Optimization: A Case Study on Community QA and Summarization

Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning

Semantically Driven Sentence Fusion: Modeling and Evaluation

Contact Info

Product

Resources

About