Context‐Aware Mixed Reality: A Learning‐Based Framework for Semantic‐Level Interaction

Abstract

Mixed reality (MR) is a powerful interactive technology for new types of user experience. We present a semantic‐based interactive MR framework that is beyond current geometry‐based approaches, offering a step change in generating high‐level context‐aware interactions. Our key insight is that by building semantic understanding in MR, we can develop a system that not only greatly enhances user experience through object‐specific behaviours, but also it paves the way for solving complex interaction design challenges. In this paper, our proposed framework generates semantic properties of the real‐world environment through a dense scene reconstruction and deep image understanding scheme. We demonstrate our approach by developing a material‐aware prototype system for context‐aware physical interactions between the real and virtual objects. Quantitative and qualitative evaluation results show that the framework delivers accurate and consistent semantic information in an interactive MR environment, providing effective real‐time semantic‐level interactions.

A Review on Mixed Reality: Current Trends, Challenges and Prospects

“…Ref. [37] proposed the use of surveys with the ANOVA test in learning applications to detect inconsistent views. Ref.…”

“…Ref. [37] proposed the use of image processing based on semantic segmentation using conditional random fields as recurrent neural networks (CRF-RNN) to provide an appropriate context for display in a material-aware platform. Ref.…”

“…In image-based approaches based on deep learning, choosing the right training data set to identify the general object of the MR system is an important point. This data provides a useful context in MR systems to improve the quality of interaction [37]. In the image-processing-based approach based on labeling with voice commands, the accuracy of the method depends on concepts such as good synchronization of voice commands with the gaze.…”

“…[69] describes the characteristics of MR systems as follows: (1) real and digital content is integrated into an application; (2) an MR application is interactive in real-time; and (3) an MR application is registered in three dimensions. Regarding these three features, various types of MR systems have been presented in various studies such as, multimodal MR [70], visuohaptic MR [71], tangible MR [72], context-aware real-time MR [37], markerless MR [43], adaptive MR [55], and multiscale MR [73]. This framework provides many benefits for users.…”

“…Visuohaptic MR needs an appropriate strategy to handle dynamic scene problems when combined with virtual object haptic devices [71]. The context-aware real-time MR needs datasets with a more general feature to obtain more context and to use complex interactions based on realistic physical-based sound propagation and physical-based rendering [37]. The use of multicamera-based tracking is an important issue that improves markerless MR [43].…”