GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems

Abstract

Automatically evaluating dialogue coherence is a challenging but high-demand ability for developing high-quality open-domain dialogue systems. However, current evaluation metrics consider only surface features or utterancelevel semantics, without explicitly considering the fine-grained topic transition dynamics of dialogue flows. Here, we first consider that the graph structure constituted with topics in a dialogue can accurately depict the underlying communication logic, which is a more natural way to produce persuasive metrics. Capitalized on the topic-level dialogue graph, we propose a new evaluation metric GRADE, which stands for Graph-enhanced Representations for Automatic Dialogue Evaluation. Specifically, GRADE incorporates both coarsegrained utterance-level contextualized representations and fine-grained topic-level graph representations to evaluate dialogue coherence. The graph representations are obtained by reasoning over topic-level dialogue graphs enhanced with the evidence from a commonsense graph, including k-hop neighboring representations and hop-attention weights. Experimental results show that our GRADE significantly outperforms other state-of-the-art metrics on measuring diverse dialogue models in terms of the Pearson and Spearman correlations with human judgements. Besides, we release a new large-scale human evaluation benchmark to facilitate future research on automatic metrics.

Towards Quantifiable Dialogue Coherence Evaluation

“…Moreover, learnable metrics encoding the semantic information have been attracting interests recently, which are trained in a supervised manner with large-scale human-annotated data, such as ADEM (Lowe et al, 2017), or trained in an unsupervised manner with automatically constructed data, such as RUBER (Tao et al, 2018) and BERT-RUBER (Ghazarian et al, 2019). Furthermore, the recently proposed coherence metric, GRADE (Huang et al, 2020), introduces the graph information of dialogue topic transitions and achieves the current state-of-the-art results. Note that these learnable metrics are trained in a two-level training objective to separate the coherent dialogues from the incoherent ones, while our QuantiDCE models the task in a multi-level setting which is closer to the actual human rating.…”

“…Therefore, in this work, we set the number of coherence levels L = 3 where the pairs containing the random responses, the adversarial responses and the reference responses respectively belong to the levels from 1 to 3. As to the fine-tuning data, we use the DailyDialog human judgement dataset, denoted as DailyDialogEVAL, which is a subset of the adopted evaluation benchmark (Huang et al, 2020), with 300 human rating data in total, and randomly split the data into training (90%) and validation (10%) sets. Implementation Details.…”

“…Baseline Metrics. We compare the metric model trained by our QuantiDCE with eight popular automatic dialogue metrics, including three lexical word-overlap metrics: BLEU (Papineni et al, 2002), ROUGE (Lin, 2004) and ME-TEOR (Banerjee and Lavie, 2005), one semantic word-overlap metric, BERTScore , and four learnable metrics: ADEM (Lowe et al, 2017), BERT-RUBER (Ghazarian et al, 2019), BLEURT (Sellam et al, 2020) and GRADE (Huang et al, 2020).…”

Towards Making the Most of Dialogue Characteristics for Neural Chat Translation

“…Context-Aware NMT. In a sense, chat MT can be viewed as a special case of context-aware MT that has many related studies (Gong et al, 2011;Jean et al, 2017;Wang et al, 2017b;Zheng et al, 2020;Yang et al, 2019;Kang et al, 2020;Ma et al, 2020). Typically, they resort to extending conventional NMT models for exploiting the context.…”

Improving Unsupervised Dialogue Topic Segmentation with Utterance-Pair Coherence Scoring

“…We follow previous work (Wang et al, 2017;Xu et al, 2019;Huang et al, 2020) to optimize the utterance-pair coherence scoring model (described in Section 3.2) with marginal ranking loss. Formally, the coherence scoring model CS receives two utterances (u 1 , u 2 ) as input and return the coherence score c = CS(u 1 , u 2 ), which reflects the topical relevance of this pair of utterances.…”