Evaluating spoken dialogue agents with PARADISE: Two case studies

Walker, Marilyn A.; Litman, Diane; Kamm, Candace; Abella, Alicia

doi:10.1006/csla.1998.0110

Cited by 109 publications

(103 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…One key issue is that spoken dialogue systems are not good at determining whether the communication is going well or whether communication problems arose (e.g., due to poor speech recognition or false default assumptions). The occurrence of problems negatively affects user satisfaction (Walker et al, 1998), but also has an impact on the way users communicate with the 0167-6393/$ -see front matter Ó 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.specom.…”

Section: Introductionmentioning

confidence: 99%

Problem detection in human–machine interactions based on facial expressions of users

Barkhuysen¹,

Krahmer²,

Swerts³

2005

Speech Communication

View full text Add to dashboard Cite

This paper describes research into audiovisual cues to communication problems in interactions between users and a spoken dialogue system. The study consists of two parts. First, we describe a series of three perception experiments in which subjects are offered film fragments (without any dialogue context) of speakers interacting with a spoken dialogue system. In half of these fragments, the speaker is or becomes aware of a communication problem. Subjects have to determine by forced choice which are the problematic fragments. In all three tests, subjects are capable of performing this task to some extent, but with varying levels of correct classifications. Second, we report results of an observational analysis in which we first attempt to relate the perceptual results to visual features of the stimuli presented to subjects, and second to find out which visual features actually are potential cues for error detection. Our major finding is that more problematic contexts lead to more dynamic facial expressions, in line with earlier claims that communication errors lead to marked speaker behaviour. We conclude that visual information from a userÕs face is potentially beneficial for problem detection.

show abstract

Section: Introductionmentioning

confidence: 99%

Problem detection in human–machine interactions based on facial expressions of users

Barkhuysen¹,

Krahmer²,

Swerts³

2005

Speech Communication

View full text Add to dashboard Cite

show abstract

“…In addition, at the end of the testing session, each participant was asked to fill in a relevant questionnaire. Table 2 shows example questions (based on [6,16,22,29], among others) used as subjective evaluation criteria and the usability aspect they correspond to. Participants were asked to assess the two systems on a Likert scale from 1 to 5.…”

Section: Usability Evaluation: Procedure Results and Discussionmentioning

confidence: 99%

Spoken Dialogue Interfaces: Integrating Usability

Spiliotopoulos

Stavropoulou

2009

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Usability is a fundamental requirement for natural language interfaces. Usability evaluation reflects the impact of the interface and the acceptance from the users. This work examines the potential of usability evaluation in terms of issues and methodologies for spoken dialogue interfaces along with the appropriate designer-needs analysis. It unfolds the perspective to the usability integration in the spoken language interface design lifecycle and provides a framework description for creating and testing usable content and applications for conversational interfaces. Main concerns include the problem identification of design issues for usability design and evaluation, the use of customer experience for the design of voice interfaces and dialogue, and the problems that arise from real-life deployment. Moreover it presents a real-life paradigm of a hands-on approach for applying usability methodologies in a spoken dialogue application environment to compare against a DTMF approach. Finally, the scope and interpretation of results from both the designer and the user standpoint of usability evaluation are discussed.

show abstract

“…This field can be considered to be in an initial phase of development. PARADISE (PARAdigm for DIalogue Evaluation System) is the most widely proposed methodology to perform a global evaluation of a dialog system (Walker et al, 1998) (Dybkjaer et al, 2004). This methodology combines different measures regarding task success, dialog efficiency and dialog quality in a single function that measures the yield of the system in direct correlation with user satisfaction.…”

Section: Related Work Evaluation Of Dialog Systemsmentioning

confidence: 99%