Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems

Abstract

The lack of time-efficient and reliable evaluation methods hamper the development of conversational dialogue systems (chatbots). Evaluations requiring humans to converse with chatbots are time and cost-intensive, put high cognitive demands on the human judges, and yield low-quality results. In this work, we introduce Spot The Bot, a cost-efficient and robust evaluation framework that replaces human-bot conversations with conversations between bots. Human judges then only annotate for each entity in a conversation whether they think it is human or not (assuming there are humans participants in these conversations). These annotations then allow us to rank chatbots regarding their ability to mimic the conversational behavior of humans. Since we expect that all bots are eventually recognized as such, we incorporate a metric that measures which chatbot can uphold human-like behavior the longest, i.e., Survival Analysis. This metric has the ability to correlate a bot's performance to certain of its characteristics (e.g., fluency or sensibleness), yielding interpretable results. The comparably low cost of our framework allows for frequent evaluations of chatbots during their evaluation cycle. We empirically validate our claims by applying Spot The Bot to three domains, evaluating several stateof-the-art chatbots, and drawing comparisons to related work. The framework is released as a ready-to-use tool.

Towards Automated Psychotherapy via Language Modeling

“…Similarly, as shown in figure 4, the model was able to meet or exceed human performance for "Spot the Bot" [5] Turing evaluations for 67.16% of the validation set. In fact, 3 response data-points even coded for human responses being most likely generated, and in two of which the model achieved a higher score.…”

“…After training, the model was evaluated based on variants of two sets of published metrics. Singleblinded, independently coded responses show that the model was able to synthesize utterances at or above human level 59.7% and 67.16% of the time for RQI (response quality index) [15] and a comparative Turing test ("Spot the Bot", [5]) respectively for a test set of 134 validation prompt/response pairs. Of the samples where human responses did outperform synthesized responses, only 17.9% and 20.9% of the subset did so significantly.…”

“…Second, an altered measurement of Response Quality Index (RQI) similar to that proposed by Xiao, et al [5] The final "quality factor" result from this second part of validation is, as is similar to Xiao, et al, the product of the scores from all three evaluated factors; formally:…”

“…In order to address this, a two-part evaluation rubric was created corresponding to previously published work [5] [15]. The generated prompt/response pairs and their corresponding human-written validation prompt/response pair were both scored upon the devised rubric in a single-blinded fashion.…”

“…First, a single-blinded adaptation of the Turing test named "Spot The Bot" as devised by Deriu, et al [5] was administered. In this segment, blinded human evaluators were tasked with scoring each prompt/response pair on a scale between 1 to 3 -with 1 representing "Likely Generated" to 3 representing "Likely Human-Written."…”