Empirically Evaluating an Adaptable Spoken Dialogue System

Abstract

Recent technological advances have made it possible to build real-time, interactive spoken dialogue systems for a wide variety of applications. However, when users do not respect the limitations of such systems, performance typically degrades. Although users differ with respect to their knowledge of system limitations, and although different dialogue strategies make system limitations more apparent to users, most current systems do not try to improve performance by adapting dialogue behavior to individual users. This paper presents an empirical evaluation of TOOT, an adaptable spoken dialogue system for retrieving train schedules on the web. We conduct an experiment in which 20 users carry out 4 tasks with both adaptable and non-adaptable versions of TOOT, resulting in a corpus of 80 dialogues. The values for a wide range of evaluation measures are then extracted from this corpus. Our results show that adaptable TOOT generally outperforms non-adaptable TOOT, and that the utility of adaptation depends on TOOT's initial dialogue strategies. ⋆ We thank J. Chu-Carroll, C. Kamm, D. Lewis, M. Walker, and S. Whittaker for helpful comments.

Spoken Dialogue Interfaces: Integrating Usability

“…These aspects can be used as the basis for usability evaluation strategies. Many frameworks and methodologies have been developed and used for evaluation of spoken dialogue systems in recent works [8,9,13,15,17,18,21,24,30].…”

Detecting and Exploiting User Familiarity in Natural Language Human-computer Dialogue

“…2.1 Static and dynamic user models Due to significant variations in human-computer dialogue system performances according to the type of user and the type of dialogue, as is the case with more general interactive systems such as human-computer interaction, user modelling has become a critical aspect of designing systems and services (Litman & Pan, 2002). Like the knowledge pertaining to users that is contained in the context knowledge bases of human-computer dialogue systems, user models are constructed from both a priori and dynamically acquired knowledge about users.…”

“…This prediction may concern problematic situations, such as those caused by speech recognition errors. Human-computer dialogue systems are then designed to avoid this situation (Litman & Pan, 2002). Prediction can be used by human-computer dialogue systems to anticipate a user's behaviour and minimise his or her contribution.…”

Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system

“…For example, in Litman and Pan (2002), dialogue strategies were automatically adapted dynamically based on repeated speech recognition errors. The adaptive system outperformed the non-adaptive system for novice users by significantly increasing the task completion rate.…”