D-Score: Holistic Dialogue Evaluation Without Reference

Abstract

In artistic gymnastics, difficulty score or D-score is used for judging performance. Starting from zero, an athlete earns points from different aspects such as composition requirement, difficulty, and connection between moves. The final score is a composition of the quality of various performance indicators. Similarly, when evaluating dialogue responses, human judges generally follow a number of criteria, among which language fluency, context coherence, logical consistency, and semantic appropriateness are on top of the agenda. In this paper, we propose an automatic dialogue evaluation framework called D-score that resembles the way gymnastics is evaluated. Following the four human judging criteria above, we devise a range of evaluation tasks and model them under a multi-task learning framework. The proposed framework, without relying on any human-written reference, learns to appreciate the overall quality of human-human conversations through a representation that is shared by all tasks without over-fitting to individual task domain. We evaluate D-score by performing comprehensive correlation analyses with human judgement on three dialogue evaluation datasets, among which two are from past DSTC series, and benchmark against state-of-the-art baselines. D-score not only outperforms the best baseline by a large margin in terms of system-level Spearman correlation but also represents an important step towards explainable dialogue scoring.

DynaEval: Unifying Turn and Dialogue Level Evaluation

“…1 https://github.com/e0397123/DynaEval Commonly used static metrics, such as BLEU (Papineni et al, 2002), ME-TEOR (Denkowski and Lavie, 2014) and ROUGE (Lin, 2004), correlate poorly with human judgements (Liu et al, 2016) rendering them unsuitable for dialogue evaluation. While some recent automatic dialogue evaluation metrics (Ghazarian et al, 2019;Mehri and Eskenazi, 2020b;Zhang et al, 2021b) demonstrate strong correlations with human judgement at the turn-level, they only focus on context-response pairs without explicitly modeling the interaction over an entire dialogue. To perform dialogue-level evaluation, we need to rely on the aggregation of turn-level scores over the dialogue as a proxy for a dialogue-level score.…”

Survey of Hallucination in Natural Language Generation

“…Two possible types of inconsistency occur in open-domain dialogue generation: (1) inconsistency among the system utterances such as when the system contradicts its previous utterance; (2) inconsistency with some external source, such as factually incorrect utterances. Whereas the first type is described using the term "consistency" [100,186,208] or "coherence" [11,39], people recently start to call the second type "hallucination" [122,152]. Selfinconsistency can be considered as an intrinsic hallucination problem, while the external inconsistency involves both intrinsic and extrinsic hallucinations, depending on the reference source.…”

“…Model-based Metric. Recently, several works have proposed evaluation metrics for measuring consistency, such as using natural language inference (NLI) [39,186], training learnable evaluation metrics [208], or releasing additional test set for coherence [11]. For the KGD task, Dziri et al [41] propose the BEGIN benchmark, which consists of samples taken from Dinan et al [31] with additional human annotation and a new classification task extending the Natural Language Inference (NLI) paradigm.…”

Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges

“…HolisticEval (Pang et al, 2020) adopts different models for evaluating several qualities of dialog: context coherence, language fluency, response diversity, and logical self-consistency. D-score (Zhang et al, 2021d) adopts a single multitask model for evaluating various dialog qualities including context coherence, language fluency, logical self-consistency, and semantic appropriateness. Deep AM-FM (Zhang et al, 2021c) measures both semantic similarity and response fluency and the PARADISE-style model of (Walker et al, 2021) uses both predicted user ratings and dialog length.…”