Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining

Abstract

There is an increasing focus on model-based dialog evaluation metrics such as ADEM, RUBER, and the more recent BERT-based metrics. These models aim to assign a high score to all relevant responses and a low score to all irrelevant responses. Ideally, such models should be trained using multiple relevant and irrelevant responses for any given context. However, no such data is publicly available, and hence existing models are usually trained using a single relevant response and multiple randomly selected responses from other contexts (random negatives). To allow for better training and robust evaluation of model-based metrics, we introduce the DailyDialog++ dataset, consisting of (i) five relevant responses for each context and (ii) five adversarially crafted irrelevant responses for each context. Using this dataset, we first show that even in the presence of multiple correct references, n-gram based metrics and embedding based metrics do not perform well at separating relevant responses from even random negatives. While model-based metrics perform better than n-gram and embedding based metrics on random negatives, their performance drops substantially when evaluated on adversarial examples. To check if large scale pretraining could help, we propose a new BERT-based evaluation metric called DEB, which is pretrained on 727M Reddit conversations and then finetuned on our dataset. DEB significantly outperforms existing models, showing better correlation with human judgments and better performance on random negatives (88.27% accuracy). However, its performance again drops substantially when evaluated on adversarial responses, thereby highlighting that even large-scale pretrained evaluation models are not robust to the adversarial examples in our dataset. The dataset 1 and code 2 are publicly available.

Representation Learning for Conversational Data using Discourse Mutual Information Maximization

“…In terms of various dialog understanding tasks, our models achieve state-of-theart performances in several tasks (absolute improvements up to 8.6% in task accuracies) and perform consistently well across a variety of dialog understanding tasks at all scales, whereas baseline models usually have a rather imbalanced performance across tasks. Our models show the most promising performance on the DailyDialog++ (Sai et al 2020) dialog evaluation task.…”

“…ConveRT (Henderson et al 2020) is trained on the response retrieval task using Reddit threads. DEB or Dialog Evaluation using BERT (Sai et al 2020) is a model based on extended pretraining of the BERT architecture using Reddit data. DialogRPT (Gao et al 2020), on the other hand, is pretrained to predict human-feedback (e.g., upvotes and downvotes) on comments to Reddit threads.…”

“…In Table 4, we show sample predictions from the DailyDia-log++/cross task (Sai et al 2020). As DialoGPT has the best performance on this task among the baselines, we choose it for error analysis.…”

“…Our formulation for pretraining objective using DMI gives a possible explanation for these observations. In this work, we consider the following downstream tasks requiring knowledge of different kinds of conversational features in a text: Dialog Act Classification (Stolcke et al 1998), Intent Detection (Casanueva et al 2020), Emotion Detection (Welivita and Pu 2020), Conversational Reasoning (Cui et al 2020), and Dialog Evaluation (Sai et al 2020).…”

Krathu-500: Post-Comments Thai Corpus

“…The conversational dataset has variety of uses, e.g., improving the dialogue evaluation [2], emotion recognition [3], or use as dataset for study about personality and demographic of people on the Internet [4].…”