Reference and Document Aware Semantic Evaluation Methods for Korean Language Summarization

Lee, Dongyub; Shin, Myeong Cheol; Whang, Taesun; Cho, Seungwoo; Ko, Byeongil; Lee, Daniel; Kim, EungGyun; Jo, Jaechoon

doi:10.18653/v1/2020.coling-main.491

Cited by 8 publications

(8 citation statements)

References 39 publications

(38 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, languages in which ngram-based metrics perform poorly due to the language's structure (e.g. Korean (Lee et al, 2020)) would benefit the most from our approach).…”

Section: Resultsmentioning

confidence: 99%

Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation

Rebuffel¹,

Scialom²,

Soulier³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

QUESTEVAL is a reference-less metric used in text-to-text tasks, that compares the generated summaries directly to the source text, by automatically asking and answering questions. Its adaptation to Data-to-Text tasks is not straightforward as it requires multimodal Question Generation and Answering systems on the considered tasks, which are seldom available. To this purpose, we propose a method to build synthetic multimodal corpora enabling to train multimodal components for a data-QuestEval metric. The resulting metric is reference-less and multimodal; it obtains state-of-the-art correlations with human judgment on the WebNLG and WikiBio benchmarks. We make data-QUESTEVAL's code and models available for reproducibility purpose, as part of the QUESTEVAL project. 1

show abstract

“…In particular, languages in which ngram-based metrics perform poorly due to the language's structure (e.g. Korean (Lee et al, 2020)) would benefit the most from our approach).…”

Section: Resultsmentioning

confidence: 99%

Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation

Rebuffel¹,

Scialom²,

Soulier³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…This has shown to be effective in various tasks including review reading comprehension (Xu et al 2019) and SuperGLUE (Wang et al 2019a). Existing works on multiturn response selection (Whang et al 2020;Gu et al 2020;Humeau et al 2020) also adapted this post-training approach and obtained state-of-the-art results. We also employ this post-training method in this work and show its effectiveness in improving performance.…”

Section: Proposed Methods Language Models For Response Selectionmentioning

confidence: 99%

“…Training Response Selection Models Following several researches based on contextual language models for multi-turn response selection (Whang et al 2020;Lu et al 2020;Gu et al 2020), a pointwise approach is used to learn a cross-encoder that receives both dialog context and response simultaneously. Suppose that a dialog agent is given a dialog dataset…”

Section: Proposed Methods Language Models For Response Selectionmentioning

confidence: 99%

“…Existing works (Wu et al 2017;Zhou et al 2018;Tao et al 2019a;Yuan et al 2019) have studied utterance-response matching based on attention mechanisms including selfattention (Vaswani et al 2017). Most recently, as pre-trained language models (e.g., BERT (Devlin et al 2019), RoBERTa (Liu et al 2019), and ELECTRA (Clark et al 2020)) have achieved substantial performance improvements in diverse NLP tasks, multi-turn response selection also has been resolved using such language models (Whang et al 2020;Lu et al 2020;Gu et al 2020;Humeau et al 2020).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Do Response Selection Models Really Know What’s Next? Utterance Manipulation Strategies for Multi-turn Response Selection

Whang¹,

Lee

Oh³

et al. 2021

AAAI

Self Cite

View full text Add to dashboard Cite

In this paper, we study the task of selecting the optimal response given a user and system utterance history in retrieval-based multi-turn dialog systems. Recently, pre-trained language models (e.g., BERT, RoBERTa, and ELECTRA) showed significant improvements in various natural language processing tasks. This and similar response selection tasks can also be solved using such language models by formulating the tasks as dialog--response binary classification tasks. Although existing works using this approach successfully obtained state-of-the-art results, we observe that language models trained in this manner tend to make predictions based on the relatedness of history and candidates, ignoring the sequential nature of multi-turn dialog systems. This suggests that the response selection task alone is insufficient for learning temporal dependencies between utterances. To this end, we propose utterance manipulation strategies (UMS) to address this problem. Specifically, UMS consist of several strategies (i.e., insertion, deletion, and search), which aid the response selection model towards maintaining dialog coherence. Further, UMS are self-supervised methods that do not require additional annotation and thus can be easily incorporated into existing approaches. Extensive evaluation across multiple languages and models shows that UMS are highly effective in teaching dialog consistency, which leads to models pushing the state-of-the-art with significant margins on multiple public benchmark datasets.

show abstract

“…5 . Token level metrics like ROUGE or BERTScore are not suited to all the language morphologies [36]. We want BEAMetrics to measure the multilingual ability of a metric.…”

Section: Design Principlesmentioning

confidence: 99%

BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation

Scialom,

Hill

2021

Preprint

View full text Add to dashboard Cite

Natural language processing (NLP) systems are increasingly trained to generate open-ended text rather than classifying between responses. This makes research on evaluation metrics for generated language -functions that score system output given the context and/or human reference responses -of critical importance. However, different metrics have different strengths and biases, and reflect human intuitions better on some tasks than others. There is currently no simple, unified way to compare, analyse or evaluate metrics across a representative set of tasks. Here, we describe the Benchmark to Evaluate Automatic Metrics (BEAMetrics), a resource to make research into new metrics itself easier to evaluate. BEAMetrics users can quickly compare existing and new metrics with human judgements across a diverse set of tasks, quality dimensions (fluency vs. coherence vs. informativeness etc), and languages. As generation experts might predict, BEAMetrics reveals stark taskdependent differences between existing metrics, and consistently poor performance on tasks with complex answer spaces or high reliance on general knowledge. While this analysis highlights a critical issue facing current research practice, BEAMetrics also contribute to its resolution by facilitating research into better metrics -particularly those that can account for the complex interaction between context and general knowledge inherent to many modern NLP applications. 1

show abstract

Reference and Document Aware Semantic Evaluation Methods for Korean Language Summarization

Cited by 8 publications

References 39 publications

Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation

Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation

Do Response Selection Models Really Know What’s Next? Utterance Manipulation Strategies for Multi-turn Response Selection

BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation

Contact Info

Product

Resources

About