DynaEval: Unifying Turn and Dialogue Level Evaluation

Zhang, Chen; Chen, Yiming; D’Haro, Luis Fernando; Zhang, Yan; Friedrichs, Thomas; Lee, Grandee; Li, Haizhou

doi:10.18653/v1/2021.acl-long.441

Cited by 23 publications

(40 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For system development, we rely on ADMs for model design, hyperparameter tuning and system benchmarking (Yeh, Eskenazi, and Mehri 2021). The current trend of opendomain ADMs is shifting from the reference-based approach towards the model-based approach that is referencefree (Mehri and Eskenazi 2020a;Zhang et al 2021a). In many ADM solutions, we predict the relatedness between a dialogue context and the generated responses by training a discriminative network to distinguish the original response from negative samples in a self-supervised fashion.…”

Section: Related Work Dialogue Evaluation Metricsmentioning

confidence: 99%

MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation

Zhang

D’Haro

Friedrichs³

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

Chatbots are designed to carry out human-like conversations across different domains, such as general chit-chat, knowledge exchange, and persona-grounded conversations. To measure the quality of such conversational agents, a dialogue evaluator is expected to conduct assessment across domains as well. However, most of the state-of-the-art automatic dialogue evaluation metrics (ADMs) are not designed for multi-domain evaluation. We are motivated to design a general and robust framework, MDD-Eval, to address the problem. Specifically, we first train a teacher evaluator with human-annotated data to acquire a rating skill to tell good dialogue responses from bad ones in a particular domain and then, adopt a self-training strategy to train a new evaluator with teacher-annotated multi-domain data, that helps the new evaluator to generalize across multiple domains. MDD-Eval is extensively assessed on six dialogue evaluation benchmarks. Empirical results show that the MDD-Eval framework achieves a strong performance with an absolute improvement of 7% over the state-of-the-art ADMs in terms of mean Spearman correlation scores across all the evaluation benchmarks.

show abstract

Section: Related Work Dialogue Evaluation Metricsmentioning

confidence: 99%

MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation

Zhang

D’Haro

Friedrichs³

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…Discourse coherence is a broad area of research and in dialog evaluation, we try to assess coherence at different granularity. One is the coherence of the entire dialog flow (Zhang et al, 2021a) and the other is the local coherence at the turn-level, i.e., context-response coherence (Cervone and Riccardi, 2020). In our study, we focus on the local coherence assessment and adopt a simple metric (CoSim) to evaluate the coherence between the context, p, and the response, h:…”

Section: Context-response Coherencementioning

confidence: 99%

Investigating the Impact of Pre-trained Language Models on Dialog Evaluation

Zhang¹,

D’Haro²,

Chen³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, there is a surge of interest in applying pre-trained language models (Pr-LM) in automatic open-domain dialog evaluation. Pr-LMs offer a promising direction for addressing the multi-domain evaluation challenge. Yet, the impact of different Pr-LMs on the performance of automatic metrics is not wellunderstood. This paper examines 8 different Pr-LMs and studies their impact on three typical automatic dialog evaluation metrics across three different dialog evaluation benchmarks. Specifically, we analyze how the choice of Pr-LMs affects the performance of automatic metrics. Extensive correlation analyses on each of the metrics are performed to assess the effects of different Pr-LMs along various axes, including pre-training objectives, dialog evaluation criteria, model size, and cross-dataset robustness. This study serves as the first comprehensive assessment of the effects of different Pr-LMs on automatic dialog evaluation.

show abstract

“…A solution for better automatic evaluation methods is to train reference-free evaluators that learn how to assess the generated responses given dialog contexts from different aspects such as relevancy (Tao et al, 2018;Ghazarian et al, 2019;Lan et al, 2020), engagement (Ghazarian et al, 2020), fluency (Zhang et al, 2021b;Pang et al, 2020), contradiction (Pang et al, 2020;Nie et al, 2021) amongst others. It is also important to get some holistic evaluation at the dialog level in order to assess the dialogs as a whole (Zhang et al, 2021a;Mehri and Eskenazi, 2020;Finch et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation

Ghazarian¹,

Hedayatnia²,

Papangelis³

et al. 2022

Preprint

View full text Add to dashboard Cite

Accurate automatic evaluation metrics for open-domain dialogs are in high demand. Existing model-based metrics for system response evaluation are trained on human annotated data, which is cumbersome to collect. In this work, we propose to use information that can be automatically extracted from the next user utterance, such as its sentiment or whether the user explicitly ends the conversation, as a proxy to measure the quality of the previous system response. This allows us to train on a massive set of dialogs with weak supervision, without requiring manual system turn quality annotations. Experiments show that our model is comparable to models trained on human annotated data. Furthermore, our model generalizes across both spoken and written opendomain dialog corpora collected from real and paid users.

show abstract

DynaEval: Unifying Turn and Dialogue Level Evaluation

Cited by 23 publications

References 47 publications

MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation

MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation

Investigating the Impact of Pre-trained Language Models on Dialog Evaluation

What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation

Contact Info

Product

Resources

About