Evaluation of spoken multimodal conversation

Bernsen, Niels Ole; Dybkjær, Laila

doi:10.1145/1027933.1027941

Cited by 21 publications

(16 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such a method (questionnaire/interview) was also used by Bernsen and Dybkjaer in their experiments on NICE -a system for spoken and gesture interaction with life-like fairytale author Hans Christian Andersen [25]. The content of questions was slightly different from those we used, as the embodiment of the system and the usage of gestures also needed to be addressed.…”

Section: Discussionmentioning

confidence: 99%

“…or "What was good about your interaction with the system?" [25]). The biggest difference between our and Bernsen and Dybkjaer's evaluation was qualitative -e. g., the answers in their experiment were given freely by the users, without any quantitative scale.…”

Section: Discussionmentioning

confidence: 99%

“…[25]) evaluation questionnaires do not include any scales and require the users to answer the questions freely. In our research though we decided to use the scales, as it is easier to interpret and compare numbers than written impressions.…”

Section: Scalesmentioning

confidence: 99%

See 2 more Smart Citations

Evaluating Subjective Aspects of Hci on an Example of a Non-Task Oriented Conversational System

Dybała

Ptaszyński

Rzepka

et al. 2010

Int. J. Artif. Intell. Tools

View full text Add to dashboard Cite

The evaluation of subjective aspects of HCI, such as human-likeness, likeability or users' emotions towards computers is still quite a neglected issue, especially in the field of non-task oriented conversational systems (chatterbots). In this paper we try to bridge this gap by proposing a new methodology of evaluation. The methods presented were tested in our research on humor-equipped chatterbots. We describe them in details, discuss their drawbacks and usability. In one of the presented methods we used an emotiveness analysis system, which itself can be considered an AI tool, as it was used to detect users' emotions towards conversational systems, and to perform their automatic evaluation. We also propose some methods that we have not used yet, which, however, * Corresponding author: paweldybala@media.eng.hokudai.ac.jp (Pawel Dybala) Pawel Dybala, Michal Ptaszynski, Rafal Rzepka, and Kenji Araki 2 seem applicable in this field, such as brain scanning techniques. Finally, we give some ideas that should be addressed in the future.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluating Subjective Aspects of Hci on an Example of a Non-Task Oriented Conversational System

Dybała

Ptaszyński

Rzepka

et al. 2010

Int. J. Artif. Intell. Tools

View full text Add to dashboard Cite

show abstract

“…The high-level theory of conversation underlying Andersen's conversational behaviour is derived from analyses of social conversations aimed at making new friends, emphasising common ground, expressive story-telling, rhapsodic topic shifts, balance of interlocutor ''expertise'' (stories to tell), etc. [2]. When Andersen is alone in his study, he goes about his work, thinking, meandering in locomotion, looking out at the streets of Copenhagen, etc.…”

Section: Interacting With Andersenmentioning

confidence: 99%

Fusion of children's speech and 2D gestures when conversing with 3D characters

et al. 2006

View full text Add to dashboard Cite

is an open access repository that collects the work of Arts et Métiers ParisTech researchers and makes it freely available over the web where possible. AbstractMost existing multi-modal prototypes enabling users to combine 2D gestures and speech input are task-oriented. They help adult users solve particular information tasks often in 2D standard Graphical User Interfaces. This paper describes the NICE Andersen system, which aims at demonstrating multi-modal conversation between humans and embodied historical and literary characters. The target users are 10-18 years old children and teenagers. We discuss issues in 2D gesture recognition and interpretation as well as temporal and semantic dimensions of input fusion, ranging from systems and component design through technical evaluation and user evaluation with two different user groups. We observed that recognition and understanding of spoken deictics were quite robust and that spoken deictics were always used in multimodal input. We identified the causes of the most frequent failures of input fusion and suggest possible improvements for removing these errors. The concluding discussion summarises the knowledge provided by the NICE Andersen system on how children gesture and combine their 2D gestures with speech when conversing with a 3D character, and looks at some of the challenges facing theoretical solutions aimed at supporting unconstrained speech/2D gesture fusion. r

show abstract

“…The ability to handle the so-called "out-of-domain" questions is key to the lifelikeness of an agent [4], and coherence/appropriateness of the answer is important in maintaining engagement from the user [5], as user frustration can be caused otherwise. This is especially true for non-task oriented CAs, where there is no clear common user-agent goal [6], which means the user utterances are largely unbounded. As a result, common dialogue strategies in task-oriented CAs are no longer effective.…”

Section: Introductionmentioning

confidence: 99%

How Do People Talk with a Virtual Philosopher: Log Analysis of a Real-World Application

Wang

Nakatsu

2013

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Conversation with computers is an important form of human computer interaction. Inappropriately designed conversational agents can easily lead to unsatisfying user experience and even frustration, and this is especially true when the application is deployed in the real world. Currently, research on casual non-task oriented systems and our understanding in how people interact with such agents are still limited. To gain more insights on this issue, we carried out both quantitative and qualitative content analysis of conversation logs collected from a realworld application, featuring a non-task oriented conversational agent as a virtual philosopher. We construct a taxonomy of user utterances to the agent and discuss a few strategies that an agent might employ to provide a better user experience.

show abstract

Evaluation of spoken multimodal conversation

Cited by 21 publications

References 8 publications

Evaluating Subjective Aspects of Hci on an Example of a Non-Task Oriented Conversational System

Evaluating Subjective Aspects of Hci on an Example of a Non-Task Oriented Conversational System

Fusion of children's speech and 2D gestures when conversing with 3D characters

How Do People Talk with a Virtual Philosopher: Log Analysis of a Real-World Application

Contact Info

Product

Resources

About