Multimodal cooperation with the DenK system

Bunt, Harry; Ahn, Rene Rene; Beun, Robbert-Jan; Borghuis, Tijn; Overveld, C. W. A. M. van

doi:10.1007/bfb0052312

Cited by 15 publications

(7 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…57 The term virtual agent refers to the dynamic visualization of a virtual character, and if connected to a conversational model, represents a conversational agent which is "an autonomous construct that maintains its own representations of the state of the world and the conversation, and whose behavior is determined by these representations". 58 A virtual being in our work differs from Cassell's et al 59 definition of an embodied conversational agent (ECA) in that the latter represents a "computer-generated cartoon-like character" with a full body, whereas our virtual being conceptualizes a realistically "human-looking character," as targeted in research on virtual humans, 37,33,60 and only requires partial embodiment with the face. According to this taxonomy, a virtual being is a kind of virtual agent, and an ECA is a kind of virtual being (see Figure 1).…”

Section: Virtual Beingsmentioning

confidence: 96%

“…When the virtual character depicts the user as self‐presentation, for example, in computer games, it is called an avatar 57 . The term virtual agent refers to the dynamic visualization of a virtual character, and if connected to a conversational model, represents a conversational agent which is “an autonomous construct that maintains its own representations of the state of the world and the conversation, and whose behavior is determined by these representations” 58 …”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Natural conversations with a virtual being: How user experience with a current conversational AI model compares to expectations

Khvan

Choi

2023

Computer Animation & Virtual

View full text Add to dashboard Cite

The present work investigates the effect of natural conversations with virtual beings on user perceptions with a current conversational AI model (Meta's BlenderBot). To this aim, we designed a virtual being from a deep learning‐generated face and a conversational AI model acting as a virtual conversation partner in an online conferencing software and evaluated it in 11 perceptions of social attributes. Compared to prior expectations, participants perceived the virtual being as distinctly higher in warmth (engaging, empathic, and approachable) but lower in realism and credibility after 5 days of 10 min daily conversations (Study 1). Further, we explored the idea of simplifying the technical setup to reduce the technical entry barrier for such AI applications (Study 2). To this aim, we conducted several trials of fine‐tuning a small conversational model of 90 million parameters until its performance metrics improved. Testing this fine‐tuned model with users revealed that this model was not perceived differently from a large conversational model (1.4 billion parameters). In summary, our findings show that recent progress in conversational AI has added warmth‐related aspects to the user experience with virtual beings, and that fine‐tuning a conversational AI model can be effective to reduce technical complexity.

show abstract

Section: Virtual Beingsmentioning

confidence: 96%

Section: Related Workmentioning

confidence: 99%

Natural conversations with a virtual being: How user experience with a current conversational AI model compares to expectations

Khvan

Choi

2023

Computer Animation & Virtual

View full text Add to dashboard Cite

show abstract

“…The key reasoning behind its use is that human interaction is multimodal in the real world [74,75], including interactions between an individual and trainer [76]. The order and pattern by which multimodal information should be integrated has long been the subject of study in the field of human-computer interactions [77][78][79][80][81].…”

Section: Multimodal Integrationmentioning

confidence: 99%

A Survey of Multimodal Systems and Techniques for Motor Learning

Tadayon

McDaniel²,

Panchanathan

2017

J Inf Process Syst

View full text Add to dashboard Cite

This survey paper explores the application of multimodal feedback in automated systems for motor learning. In this paper, we review the findings shown in recent studies in this field using rehabilitation and various motor training scenarios as context. We discuss popular feedback delivery and sensing mechanisms for motion capture and processing in terms of requirements, benefits, and limitations. The selection of modalities is presented via our having reviewed the best-practice approaches for each modality relative to motor task complexity with example implementations in recent work. We summarize the advantages and disadvantages of several approaches for integrating modalities in terms of fusion and frequency of feedback during motor tasks. Finally, we review the limitations of perceptual bandwidth and provide an evaluation of the information transfer for each modality.

show abstract

“…We illustrate the possible combination of basic components, general mechanisms, and contextual data into a multimodal meaning representation and exemplify the general methodology that we suggested here, by taking up a sample semantic representation derived from an initial example expressed in the ULF+ format (ULF+ is a slightly updated version of a representation language that was developed successively in the PLUS dialogue project, see Geurts and Rentier, 1993, and the multimodal DENK project; see Bunt et al, 1998;Kievit, 1998). Figure 5 shows a fragment of a (simplified) possible XML-style multimodal meaning representation of the utterance "I want to go from here to there" uttered by a speaker named Peter who points at locations on a map while speaking.…”

Section: Figure 5 Example Of Multimodal Semantic Representationmentioning

confidence: 99%

Fusion and Coordination for Multimodal Interactive Information Presentation

Kipp

Wahlster

Maybury

et al. 2005

Multimodal Intelligent Information Presentation

Self Cite

View full text Add to dashboard Cite

Users require more effective and efficient means of interaction with increasingly complex information and new interactive devices. This document summarizes the results of the international Dagstuhl Seminar on Coordination and Fusion in Multimodal Interaction that took place at Schloss Dagstuhl in Germany October 27 through November 2, 20011 . We first outline a research roadmap in the near and long term. Next we describe requirements and an abstract architecture for this class of systems. We then detail requirements for semantic representations and languages necessary to enable these systems. Finally, we describe data, annotation methodologies and tools necessary to further advance the field. We conclude with a recommended action plan for forward progress in the community.1. 0 ROADMAP Figure 1 illustrates the roadmap in the near term, from 2002-2005 for the creation of mobile, human-centered intelligent multimodal interfaces. Three "lanes" in the road identify three areas of research and development, including empirical and data driven models of multimodality, advanced methods for multimodal communication and toolkits for multimodal systems. The end of the road maps indicate the outcome in 2005, specifically multimodal corpora, computational models, and interface toolkits. Of course there are a variety of interim outcomes along with way. For multimodal corpora this includes annotated corpora of human and natural phenomena (e.g., surveillance, meeting, or broadcast news video) as well as human-machine interactions. Corpora can be used by systems for training or testing/evaluation purposes. In the methods lane, this includes developments such as multimodal mutual disambiguation, multiparty interaction, and multimodal barge in. With respect to toolkits, developments include markup standards for multimodal phenomena (e.g., for combinations of speech, gesture, and facial expressions), reusable components for multimodal analysis and generation, and tools for universal and mobile multimodal access.1 Some slides are available at www.dfki.de/~wahlster/Dagstuhl_Multi_Modality/

show abstract

Multimodal cooperation with the DenK system

Cited by 15 publications

References 11 publications

Natural conversations with a virtual being: How user experience with a current conversational AI model compares to expectations

Natural conversations with a virtual being: How user experience with a current conversational AI model compares to expectations

A Survey of Multimodal Systems and Techniques for Motor Learning

Fusion and Coordination for Multimodal Interactive Information Presentation

Contact Info

Product

Resources

About