How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

Abstract

We investigate evaluation metrics for dialogue response generation systems where supervised labels, such as task completion, are not available. Recent works in response generation have adopted metrics from machine translation to compare a model's generated response to a single target response. We show that these metrics correlate very weakly with human judgements in the non-technical Twitter domain, and not at all in the technical Ubuntu domain. We provide quantitative and qualitative results highlighting specific weaknesses in existing metrics, and provide recommendations for future development of better automatic evaluation metrics for dialogue systems.

Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

“…We also conduct an analysis of the response data from (Liu et al, 2016), where the pre-processing is standardized by removing '<first speaker>' tokens at the beginning of each utterance. The results are detailed in the supplemental material.…”

“…The most widely used metric for evaluating such dialogue systems is BLEU (Papineni et al, 2002), a metric measuring word overlaps originally developed for machine translation. However, it has been shown that BLEU and other word-overlap metrics are biased and correlate poorly with human judgements of response quality (Liu et al, 2016). There are many obvious cases where these metrics fail, as they are often incapable of considering the semantic similarity between responses (see Figure 1).…”

“…To achieve this variety, we use candidate responses from several different models. Following (Liu et al, 2016), we use the following 4 sources of candidate responses: (1) a response selected by a TF-IDF retrieval-based model, (2) a response selected by the Dual Encoder (DE) (Lowe et al, 2015), (3) a response generated using the hierarchical recurrent encoder-decoder (HRED) model (Serban et al, 2016a), and (4) human-generated responses. It should be noted that the humangenerated candidate responses are not the reference responses from a fixed corpus, but novel human responses that are different from the reference.…”

Boosting a Rule-Based Chatbot Using Statistics and User Satisfaction Ratings

“…Our goal is to take advantage of this feature not only for evaluation, but also for the system's actual design. As far as the evaluation of unsupervised response generation systems goes, this is a challenging area of research in its own right [19,18].…”

Quality Signals in Generated Stories

“…This is also the case in open-domain dialogue systems, in which common evaluation metrics like BLEU (Papineni et al, 2002) are only weakly correlated with human judgments (Liu et al, 2016). Another problem with metrics like BLEU is the dependence on a gold standard.…”