Field trials of the Italian Arise train timetable system

Baggia, Paolo; Castagneri, Giuseppe; Danieli, Morena

doi:10.1016/s0167-6393(99)00068-0

Cited by 16 publications

(13 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…First, we hope to repeat this analysis with a much larger corpus of dialogues collected via our participation in the DARPA COMMUNICATOR evaluations (Walker and Hirschman, 1999). Second, we hope to extend these techniques to dialogues collected in eld studies as well as those collected in experiments such as we report here, by applying similar techniques to those reported in (Baggia et al, 1998). Finally, we believe that much research remains to be done to develop additional metrics for predicting user satisfaction that will generalize across systems.…”

Section: Discussion and Future Workmentioning

confidence: 87%

Towards developing general models of usability with PARADISE

2000

View full text Add to dashboard Cite

The design of methods for performance evaluation is a major open research issue in the area of spoken language dialogue systems. In this paper we present the PARADISE methodology for developing predictive models of spoken dialogue performance, and then show how to evaluate the predictive power and generalizability of such models. To illustrate our methodology, we develop a number of models for predicting system usability (as measured by user satisfaction), based on the application of PARADISE to experimental data from three di erent spoken dialogue systems. We then measure the extent to which our models generalize across di erent systems, di erent experimental conditions, and di erent user populations, by testing models trained on a subset of our corpus against a test set of dialogues. Our results show that our models generalize well across our three systems, and are thus a rst approximation towards a general performance model of system usability.

show abstract

Section: Discussion and Future Workmentioning

confidence: 87%

Towards developing general models of usability with PARADISE

2000

View full text Add to dashboard Cite

show abstract

“…Deployed systems and research prototypes exist for applications such as personal email and calendars, travel and restaurant information, personal banking, and customer care. Within the last few years, several spoken dialogue systems for widely used applications have moved from research labs into commercial use (Baggia et al, 1998;Gorin et al, 1997). These applications can receive millions of calls a month.…”

Section: Introductionmentioning

confidence: 99%

“…At each stage from research prototype to deployed commercial application, the system is constantly undergoing further development. When a system is prototyped in house or first tested in the field, human subjects are often paid to use the system and give detailed feedback on task completion and user satisfaction (Baggia et al, 1998;Walker et al, 2001). Even when a system is deployed, it often keeps evolving, either because customers want to do different things with it, or because new tasks arise out of developments in the underlying application.…”

Section: Introductionmentioning

confidence: 99%

What's the trouble

Hastie

Prasad

Walker

2001

Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02

View full text Add to dashboard Cite

Spoken dialogue systems promise efficient and natural access to information services from any phone. Recently, spoken dialogue systems for widely used applications such as email, travel information, and customer care have moved from research labs into commercial use. These applications can receive millions of calls a month. This huge amount of spoken dialogue data has led to a need for fully automatic methods for selecting a subset of caller dialogues that are most likely to be useful for further system improvement, to be stored, transcribed and further analyzed. This paper reports results on automatically training a Problematic Dialogue Identifier to classify problematic human-computer dialogues using a corpus of 1242 DARPA Communicator dialogues in the travel planning domain. We show that using fully automatic features we can identify classes of problematic dialogues with accuracies from 67% to 89%.

show abstract

“…However, in the literature some authors employ the term "field test" or "field trial" to describe studies in which the interactions are carried out by users who employ the telephone network instead of a laboratory environment, even when they are following predefined scenarios. This is the case, for example, for the evaluation of the ARISE spoken dialogue system (Baggia et al, 2000), which measures the impact of a train timetable system on the working routines of human operators, and on the callers who are traditionally served by the operators. Although the authors report experiments as "field studies", in the first experiment they contacted different callers who were asked to use the system by following different scenarios, and to fill in a questionnaire where they expressed their opinions about the system performance.…”

mentioning

confidence: 99%