Towards Quantifiable Dialogue Coherence Evaluation

Abstract

Automatic dialogue coherence evaluation has attracted increasing attention and is crucial for developing promising dialogue systems. However, existing metrics have two major limitations: (a) they are mostly trained in a simplified two-level setting (coherent vs. incoherent), while humans give Likert-type multi-level coherence scores, dubbed as "quantifiable"; (b) their predicted coherence scores cannot align with the actual human rating standards due to the absence of human guidance during training. To address these limitations, we propose Quantifiable Dialogue Coherence Evaluation (QuantiDCE), a novel framework aiming to train a quantifiable dialogue coherence metric that can reflect the actual human rating standards. Specifically, QuantiDCE includes two training stages, Multi-Level Ranking (MLR) pre-training and Knowledge Distillation (KD) fine-tuning. During MLR pre-training, a new MLR loss is proposed for enabling the model to learn the coarse judgement of coherence degrees. Then, during KD fine-tuning, the pretrained model is further finetuned to learn the actual human rating standards with only very few human-annotated data. To advocate the generalizability even with limited finetuning data, a novel KD regularization is introduced to retain the knowledge learned at the pre-training stage. Experimental results show that the model trained by QuantiDCE presents stronger correlations with human judgements than the other state-of-the-art metrics. 1 * Corresponding Author. 1 The code and trained checkpoints are available at https: //github.com/James-Yip/QuantiDCE.

Dialogues Are Not Just Text: Modeling Cognition for Dialogue Coherence Evaluation

“…Baseline We compare our evaluation metrics with eleven popular automatic dialogue evaluation metrics, including three lexical word-overlap metrics: BLEU, ROUGE, and METEOR (Banerjee and Lavie 2005), five metrics that consider semantic representation: BERTScore, ADEM (Lowe et al 2017), BERT-RUBER, BLEURT, QuantiDCE (Ye et al 2021), two metrics that take into account additional information about the dialogue: DynaEval, GRADE, and Chat-GPT. Evaluation The common practice to show the effectiveness of a dialogue evaluation metric is to calculate the correlation between the model-predicted and the humanrated scores (Zhang et al 2021;Huang et al 2020).…”

“…Dialogue coherence evaluation is essential for research on open dialogue systems, which refers to the coherence and consistency of the content and structure of the dialogue (See et al 2019;Ye et al 2021). Dialogues exhibit higher coherence when the responses are fluent in the language, clear in meaning, context-sensitive, and logically tight.…”