(Re-)examination of multimodal augmented reality

Abstract

The majority of augmented reality (AR) research has been concerned with visual perception, however the move towards multimodality is imminent. At the same time, there is no clear vision of what multimodal AR is. The purpose of this position paper is to consider possible ways of examining AR other than using popular notions and definitions, in order to come to an understanding of multimodal AR. Three concepts are introduced. First, the commonly used terms 'real' and 'virtual' are redefined in terms of stimuli rather than objects, and mediated stimuli are introduced as a new category. This addition facilitates the support of, for example, sensory substitution stimuli in AR. Next, the popular definitions of AR are examined, and a new analysis is presented. Specifically, it is theorized that the core of AR is not necessarily the combination of real and virtual, but the combination of a basis and an augmentation. Such an abstraction allows the re-introduction of augmented virtuality and, more generally, mixed reality. Lastly, a classification system for different forms of multimodal mixed reality that applies this basis-augmentation model is presented.

The Supernumerary Hand Illusion in Augmented Reality

“…At a theoretical level, for example, it is common to consider a completely computer-generated environment as VR and the real world with virtual elements as AR. However, using Milgram and Kishino's definition of virtual objects [19], the study by Suzuki et al [27] would no longer be considered AR because the virtual arm is in fact not 'virtual' but 'mediated' [23], even though it is quite agreed that this experiment is indeed in AR. At a more practical level, real-time seamless integration of real and virtual elements is still very challenging, while it is still not clear what levels (or even what kind of) fidelity is required for different experiences, as demonstrated by IJsselsteijn et al [12].…”

A Review of Milgram and Kishino’s Reality-Virtuality Continuum and a Mathematical Formalization for Combining Multiple Reality-Virtuality Continua

“…From the papers that present analogies with RVC, we found contributions on multimodal AR or XR. For instance, Rosa et al [36] introduced three concepts for an understanding of multimodal AR: (1) redefinition of 'real' and 'virtual' in terms of stimuli, ( 2) a new analysis of AR based on current definitions -combination of a basis and an augmentation, instead of combining real and virtual, (3) a classification system for different forms of multimodal MR for the basisaugmentation model. When trying to define XR, Rauschnabelet al [37] employed prior work on XR and qualitative insights from XR professionals.…”

Mixed Reality Interaction Techniques

“…This indicates, that multimodality per se is not always beneficial for interaction, but needs to be carefully designed to suit the task at hand. Rosa et al [274], discussed different notions of AR and Mixed Reality as well as the role of multimodality. Wilson et al [343] used a projector-camera system mounted on a pan-tilt platform for multimodal inteaction in a physical room using a combination of speech and gestures.…”