User simulation as testing for spoken dialog systems

Abstract

We propose to use user simulation for testing during the development of a sophisticated dialog system. While the limited behaviors of the state-of-the-art user simulation may not cover important aspects in the dialog system testing, our proposed approach extends the functionality of the simulation so that it can be used at least for the early stage testing before the system reaches stable performance for evaluation involving human users. The proposed approach includes a set of evaluation measures that can be computed automatically from the interaction logs between the user simulator and the dialog system. We first validate these measures on human user dialogs using user satisfaction scores. We also build a regression model to estimate the user satisfaction scores using these evaluation measures. Then, we apply the evaluation measures on a simulated dialog corpus trained from the real user corpus. We show that the user satisfaction scores estimated from the simulated corpus are not statistically different from the real users' satisfaction scores.

Towards a Flexible User Simulation for Evaluating Spoken Dialogue Systems

“…The authors of [2] and [3] have shown that even at initial stages of SDS development usability problems can be discovered. The earlier problems can be identified, the cheaper their correction is.…”

“…Of course this is not a panacea and in practice it turns into very different performances on every particular system under test. However, up to now the generalization problem in SDS field was considered to be very complex, so the authors of [2], [6], and [7] note the appropriate solutions are still of high demand.…”

“…The earlier problems can be identified, the cheaper their correction is. The authors of [2] and [4] have recently reported that semi-automated evaluation and testing can significantly decrease the expenses for quality and usability experiments. Following this idea, a simulation of user-system interactions may help to identify usability problems early and at low costs, and it might even be possible to base a tentative evaluation of the entire systems on simulated (instead of real user) interactions.…”

“…Despite of these prospective results, user simulation is still used not very frequently, except as an assessment tool to enable various machine learning algorithms by enlarging the original learning corpus [6]. Moreover, the authors of [2] and [7] have shown that all modern user simulators in the field of SDSs are either user-or system-centric. The idea of the present research is to develop a balanced concept for a data-driven simulation of user behaviour towards SDSs, covering several typical user patterns and scalable up to several common systems.…”

Analysis of a new simulation approach to dialog system evaluation

“…Schatzman (2007b) compared interaction parameters (dialog length and task success) of the model with either handcrafted parameters or parameters learned from data, and could show that both models lead to similar results, the learned parameters performing slightly better than the handcrafted ones. A similar model was used by Ai and Weng (2008) to evaluate an SDS in the restaurant selection domain.…”