MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation

Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges

“…MAUDE (Sinha et al, 2020) is trained with Noise Contrastive Estimation (NCE) (Gutmann and Hyvärinen, 2010). DEB (Sai et al, 2020) augments training data with manually-created relevant responses and adversarial irrelevant responses and MDD-Eval (Zhang et al, 2021b) adopts a teacher model to augment dialog data across different domains to achieve better performance across domains. GRADE and DynaEval leverages a graph structure to better model the dialog.…”

“…They propose mechanisms that aggregate pairwise comparisons such as estimating the probability that a given system scores better than another system. FlowScore (Li et al, 2021b) and the use of sentence embeddings in Rodríguez-Cantelar et al (2021) model dynamic information flow in the dialog history in order to evaluate the quality of a di-alog. FBD (Xiang et al, 2021) computes the distribution-wise difference between system generated conversations and human-written conversations to evaluate performance.…”

“…Improvements to the models can come in the form of (1) architectural improvements, (2) training algorithms, or (3) inference algorithms. Recent work has explored several promising directions of improvement including using graph-based representations of the dialog , combining multiple sub-metrics (Ghazarian et al, 2020;Mehri and Eskenazi, 2020b;Phy et al, 2020), better leveraging pre-trained models (Mehri and Eskenazi, 2020a;Li et al, 2021b), proposing better training objectives, and data augmentation strategies. Improving models leveraged in evaluation metrics remains a promising direction for future research.…”

“…As such, developing fine-grained metrics is a potential means of improving interpretability. Some existing automatic evaluation methods are designed to evaluate the overall quality (Li et al, 2021b;Xiang et al, 2021) or a single dimension (Ghazarian et al, 2020), but this cannot represent all of the desired dimensions of a human decision. How can we design a unified automatic evaluation metric consisting of many dimensions which are relevant to humans?…”

“…Existing model-based metrics have been criticized for their inability to generalize to out-of-distribution data (Yeh et al, 2021;Zhang et al, 2021b;Smith et al, 2022), i.e., they are only good at evaluating dialogs that are similar to their training data (overfit). Yet, open-domain dialog systems are becoming more versatile and exhibiting novel types of skills, such as being knowledgeable, being empathetic, and carrying out human-like daily conversations (Roller et al, 2021).…”