BERTScore: Evaluating Text Generation with BERT

Zhang, Tianyi; Kishore, Varsha; Wu, Felix; Weinberger, Kilian Q.; Artzi, Yoav

doi:10.48550/arxiv.1904.09675

Cited by 326 publications

(382 citation statements)

References 73 publications

Supporting

Mentioning

379

Contrasting

Unclassified

Order By: Relevance

“…The first reward promotes the coverage of radiology domain entities with corresponding reference reports, and the second reward promotes the consistency of the generated reports with their descriptions in the reference reports. Further, they combine these reward functions with the semantic equivalence metric of BERTScore [345] that results in generated reports with better performance in terms of clinical metrics.…”

Section: Reinforcement Learning Based Approachesmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as de facto operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of the applications of Transformers in medical imaging covering various aspects, ranging from recently proposed architectural designs to unsolved issues. Specifically, we survey the use of Transformers in medical image segmentation, detection, classification, reconstruction, synthesis, registration, clinical report generation, and other tasks. In particular, for each of these applications, we develop taxonomy, identify application-specific challenges as well as provide insights to solve them, and highlight recent trends. Further, we provide a critical discussion of the field's current state as a whole, including the identification of key challenges, open problems, and outlining promising future directions. We hope this survey will ignite further interest in the community and provide researchers with an up-to-date reference regarding applications of Transformer models in medical imaging. Finally, to cope with the rapid development in this field, we intend to regularly update the relevant latest papers and their open-source implementations at https://github.com/fahadshamshad/awesome-transformers-in-medical-imaging.

show abstract

Section: Reinforcement Learning Based Approachesmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…There are many types of trained models that can be used for stylized caption evaluation. These include text classifiers trained to identify the style of a sentence, or a model like BertScore [24], which uses a trained language model to compare generated text against a reference sentence. One of the main limitations of using a trained model as an evaluation metric is that it may require retraining to use on a different dataset.…”

Section: Related Workmentioning

confidence: 99%

StyleM: Stylized Metrics for Image Captioning Built with Contrastive N-grams

Li¹,

Harrison²

2022

Preprint

View full text Add to dashboard Cite

In this paper, we build two automatic evaluation metrics for evaluating the association between a machinegenerated caption and a ground truth stylized caption: OnlyStyle and StyleCIDEr 1 . OnlyStyle can be used to score any caption versus a given style. StyleCIDEr supports scoring the similarity of two compared captions with respect to their styles. We evaluate these two metrics using three stylized captioning methods trained on the PERSONALITY-CAPTIONS and FlickrStyle10K datasets: UPDOWN, MULTI-UPDOWN, and SVinVL. We also perform a human study to explore how well each caption aligns with human judgments in similar situations.

show abstract

“…In a large and comprehensive analysis of metrics in MT, however, [29] recommends deprecating BLEU as the MT evaluation standard. They suggest using more recent metrics such as BERTScore [87] or COMET [64], that are shown to reflect human judgement significantly better.…”

Section: Evaluating Metricsmentioning

confidence: 99%

“…BERTScore [87] leverages the contextualised representation of BERT to compute the similarity between the tokens.…”

Section: Metricsmentioning

confidence: 99%

BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation

Scialom,

Hill

2021

Preprint

View full text Add to dashboard Cite

Natural language processing (NLP) systems are increasingly trained to generate open-ended text rather than classifying between responses. This makes research on evaluation metrics for generated language -functions that score system output given the context and/or human reference responses -of critical importance. However, different metrics have different strengths and biases, and reflect human intuitions better on some tasks than others. There is currently no simple, unified way to compare, analyse or evaluate metrics across a representative set of tasks. Here, we describe the Benchmark to Evaluate Automatic Metrics (BEAMetrics), a resource to make research into new metrics itself easier to evaluate. BEAMetrics users can quickly compare existing and new metrics with human judgements across a diverse set of tasks, quality dimensions (fluency vs. coherence vs. informativeness etc), and languages. As generation experts might predict, BEAMetrics reveals stark taskdependent differences between existing metrics, and consistently poor performance on tasks with complex answer spaces or high reliance on general knowledge. While this analysis highlights a critical issue facing current research practice, BEAMetrics also contribute to its resolution by facilitating research into better metrics -particularly those that can account for the complex interaction between context and general knowledge inherent to many modern NLP applications. 1

show abstract

BERTScore: Evaluating Text Generation with BERT

Cited by 326 publications

References 73 publications

Transformers in Medical Imaging: A Survey

Transformers in Medical Imaging: A Survey

StyleM: Stylized Metrics for Image Captioning Built with Contrastive N-grams

BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation

Contact Info

Product

Resources

About