Towards developing general models of usability with PARADISE

Abstract

The design of methods for performance evaluation is a major open research issue in the area of spoken language dialogue systems. In this paper we present the PARADISE methodology for developing predictive models of spoken dialogue performance, and then show how to evaluate the predictive power and generalizability of such models. To illustrate our methodology, we develop a number of models for predicting system usability (as measured by user satisfaction), based on the application of PARADISE to experimental data from three di erent spoken dialogue systems. We then measure the extent to which our models generalize across di erent systems, di erent experimental conditions, and di erent user populations, by testing models trained on a subset of our corpus against a test set of dialogues. Our results show that our models generalize well across our three systems, and are thus a rst approximation towards a general performance model of system usability.

Towards human-like spoken dialogue systems

“…The steps introduced below are all micro-evaluation in this scheme: although the evaluation context is broadened with each step, the effect of the isolated component is nevertheless what is evaluated. Although macro-evaluation of human-like systems is beyond the scope of this article, it is worth mentioning that the metrics we describe here could for example be used instead of or in parallel with the cost measures in PARADISE (Walker et al, 2000), with a resulting system-wide mapping between human-likeness, task success, and user satisfaction. Developing and evaluating a component for increased human-likeness can be described as a multi-step process, where the component is tested in an increasingly broad context.…”

“…The PARADISE evaluation scheme (Walker et al, 2000), for example, calls for a success/failure judgement to be made by a participant after each information transaction -something that could easily become very disruptive. Still, the responses need to be collected during or shortly after the interaction occurs, lest it turn into a memory test.…”

“…Much effort is also spent on minimising the cost of evaluation, for example by designing evaluation methods that will generalise over systems and that may be re-used (e.g. Dybkjaer et al, 2004;Walker et al, 2000; see also Möller et al, 2007 for an overview); by automating the evaluations (e.g. Bechet et al, 2004;; or by utilising simulations instead of users in order make low cost repeat studies (e.g.…”

Spoken Dialogue Interfaces: Integrating Usability

“…These aspects can be used as the basis for usability evaluation strategies. Many frameworks and methodologies have been developed and used for evaluation of spoken dialogue systems in recent works [8,9,13,15,17,18,21,24,30].…”

Relations between de-facto criteria in the evaluation of a spoken dialogue system

“…1 http://www.cocosda.org/ address the problem of predicting system usability and user satisfaction from measurable performance criteria. This is the case of the PARADISE framework (Walker et al, 2000a), which has become one of the reference frameworks for system evaluation.…”