Tele-Immersive Interaction with Intelligent Virtual Agents Based on Real-Time 3D Modeling

Abstract

To enable intelligent agents interacting smoothly with human users, researchers have been deploying novel interaction modalities (e.g. non-verbal cue, vision and touch) in addition to agents’ conversational skills. Models of multi-modality interaction can enhance agents’ real-time perception, cognition and reaction towards the user. In this paper we report a novel tele-immersive interaction system developed using real-time 3D modelling techniques. In such system user’s full body is reconstructed using multi-view cameras and CUDA based visual hull reconstruction algorithm. User’s mesh model is then loaded into a virtual environment for interacting with an autonomous agent. Technical details and initial results of the system are illustrated in this paper. Following that a novel interaction scenario is proposed which links the virtual agent with a remote physical robot who takes the role of mediating interactions between two geographically separated users. Finally we discuss in depth the implications of such human-agent interaction and possible future improvements and directions

A systematic literature review: Real-time 3D reconstruction method for telepresence system

“…[69] mentioned 3D display technology that has been implemented in the telepresence system can be divided into two main devices, which are projectors [62,[70][71][72][73] and head-mounted devices (HMD) [45,52,58,59,[74][75][76][77]. The projector device's 3D display technologies are an on-stage hologram, autostereoscopic, and holographic projection, while HMDs can be classified into MR headsets and VR headsets.…”

A recent review and a taxonomy for hard and soft tissue visualization‐based mixed reality

“…Two papers among the selected publications used robots as a display device. Although Zhang and Ho 49 used websites for its flexibility and communication and can be used for training. Although, it limits the frame rate and the size of the input file, some gestures are manipulated in a wrong way.…”

“…The researchers in References 16,41,and 49 used a hand with gloves, mostly monochrome gloves, to distinctly separate the hand from the real environment so that the occlusion can be removed easily. Those gloves with distinct monochrome color are preferred, so that the occlusion can be removed easily.…”

“…Terms that describe the type of the surgery or any other domain, type of anatomy used, and type of tools used (hand, gloves, needles, pens, etc) are also considered and these are common in all papers. Furthermore, References 15‐18,41,47,49,and 82 did not fall into any type of surgery. Those papers are considered for this taxonomy as image/video processing for a better view which is very important in clinical data.…”

Does learning technology need an invisibility cloak?

“…Immersive worlds clearly forefront technology, using multimedia representations of learners to enable simulations which can be highly creative and relevant to learning needs (Zhang & Ho, 2012). If we apply Bourdieu's notion of "field," a social space understood by people as involving a range of actors and factors interacting with each other, then the field involved in the learning space is inhabited by learners and teachers, but also key technologies, and it may not be appropriate to subordinate the latter, since they can become influential actors in this space.…”