Probing the Robustness of Trained Metrics for Conversational Dialogue Systems

Abstract

This paper introduces an adversarial method to stress-test trained metrics to evaluate conversational dialogue systems. The method leverages Reinforcement Learning to find response strategies that elicit optimal scores from the trained metrics. We apply our method to test recently proposed trained metrics. We find that they all are susceptible to giving high scores to responses generated by relatively simple and obviously flawed strategies that our method converges on. For instance, simply copying parts of the conversation context to form a response yields competitive scores or even outperforms responses written by humans.

Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges

“…For instance, dropping punctuation or removing certain words does not decrease the scores produced by the automated metric. On the other hand, Deriu et al (2022) showed that when an automated metric is used as a reward for reinforcement learning, the policy converges to sub optimal solutions, which are rated highly by the metric. Thus, a key future direction is to develop automatic metrics which are built to be robust against these kinds of attacks.…”