MEEP: Is this Engaging? Prompting Large Language Models for Dialogue Evaluation in Multilingual Settings

Abstract

As dialogue systems become more popular, evaluation of their response quality gains importance. Engagingness highly correlates with overall quality and creates a sense of connection that gives human participants a more fulfilling experience. Although qualities like coherence and fluency are readily measured with well-worn automatic metrics, evaluating engagingness often relies on human assessment, which is a costly and time-consuming process. Existing automatic engagingness metrics evaluate the response without the conversation history, are designed for one dataset, or have limited correlation with human annotations. Furthermore, they have been tested exclusively on English conversations. Given that dialogue systems are increasingly available in languages beyond English, multilingual evaluation capabilities are essential. We propose that large language models (LLMs) may be used for evaluation of engagingness in dialogue through prompting, and ask how prompt constructs and translated prompts compare in a multilingual setting. We provide a prompt-design taxonomy for engagingness and find that using selected prompt elements with LLMs, including our comprehensive definition of engagingness, outperforms state-of-the-art methods on evaluation of engagingness in dialogue across multiple languages. 1

Stay grounded : incorporating knowledge in open-domain dialog

“…Despite its simplicity, LLM-EVal reportedly outperforms most baselines and stateof-the-art evaluation methods, including GPTScore. MEEP (Ferron et al, 2023) is another dialogue-specific evaluation method which directly uses the generated scores. Focusing on the Engagingness of a conversation (which shows a high correlation with the majority of other commonly desired conversational attributes), they provide the LLM with a detailed and multi-faceted description of response engagingness as the "variety of response according to the context, likelihood of encouraging the other participant to respond, likelihood of encouraging a quality response from the other participant, interestingness, specificity, and likelihood of creating a sense of belonging for the other participant.".…”

“…Like any other prompt-based method, these approaches can be sensitive to the structure and content of the provided instruction, including the descriptions, examples and even the score range Ferron et al (2023); Lin and Chen (2023). Nonetheless, the fact that human-aligned LLMs can follow instructions and provide competitive assessments of arbitrary dialogue features is a significant achievement for the field.…”