Multimodal Human Machine Interactions in Virtual and Augmented Reality

Abstract

Abstract. Virtual worlds are developing rapidly over the Internet. They are visited by avatars and staffed with Embodied Conversational Agents (ECAs). An avatar is a representation of a physical person. Each person controls one or several avatars and usually receives feedback from the virtual world on an audio-visual display. Ideally, all senses should be used to feel fully embedded in a virtual world. Sound, vision and sometimes touch are the available modalities. This paper reviews the technological developments which enable audio-visual interactions in virtual and augmented reality worlds. Emphasis is placed on speech and gesture interfaces, including talking face analysis and synthesis.

Interactions in Augmented and Mixed Reality: An Overview

“…They further classified this method in identification, verification, detection, segmentation, clustering, and diarization and proposed issues regarding variability, insufficient data, background noise, and adversarial attacks. Chollet et al [129] reviewed technological developments in augmented reality worlds, emphasizing speech and gesture interfaces. They stressed that speaker verification could be used for authentication purposes before starting a dialogue, for example, regarding a bank transfer in a virtual world.…”

Needs and challenges in human computer interaction for processing social emotional information

“…Given the complexity and the multimodal nature of the phenomenon, there has been a branching of engineering approaches toward the improvement and development of automatic video-audio processing, detection and synthesis techniques [23,61] with the goal of developing advanced mathematical models and algorithms for encoding/decoding emotional states from faces [27,65,102,130], speech [2,5,6,11,131] and/or body movements [16,24,77,79]. In order to succeed, the above research lines are seeking for deeper investigations and analyses of human interactional behaviors.…”