DeepMix: A Real-time Adaptive Virtual Content Registration System with Intelligent Detection

Guan, Yongjie; Hou, Xueyu; Zhang, Sheng

doi:10.1109/infocomwkshps51825.2021.9484583

Cited by 1 publication

(1 citation statement)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mobile headsets such as Microsoft HoloLens [44] and Magic Leap One [42] bring numerous opportunities to enable truly immersive augmented/mixed reality (AR/MR). To offer the best quality of experience (QoE), real-time, interactive AR/MR should be able to perceive and understand the surrounding environment in 3D for seamlessly blending virtual and physical objects [10,24]. With recent advances in 3D data capturing devices such as LiDAR and depth cameras, the computer vision (CV) community has developed several 3D object detection algorithms [12,35,51,58,63,68,73] by leveraging deep neural networks (DNNs).…”

Section: Introductionmentioning

confidence: 99%

DeepMix

Guan

Hou

et al. 2022

Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services

Self Cite

View full text Add to dashboard Cite

Mobile headsets should be capable of understanding 3D physical environments to offer a truly immersive experience for augmented/mixed reality (AR/MR). However, their small form-factor and limited computation resources make it extremely challenging to execute in real-time 3D vision algorithms, which are known to be more compute-intensive than their 2D counterparts. In this paper, we propose DeepMix, a mobility-aware, lightweight, and hybrid 3D object detection framework for improving the user experience of AR/MR on mobile headsets. Motivated by our analysis and evaluation of state-of-the-art 3D object detection models, DeepMix intelligently combines edge-assisted 2D object detection and novel, on-device 3D bounding box estimations that leverage depth data captured by headsets. This leads to low end-to-end latency and significantly boosts detection accuracy in mobile scenarios. A unique feature of DeepMix is that it fully exploits the mobility of headsets to fine-tune detection results and boost detection accuracy. To the best of our knowledge, DeepMix is the first 3D object detection that achieves 30 FPS (i.e., an end-to-end latency much lower than the 100 ms stringent requirement of interactive AR/MR). We implement a prototype of DeepMix on Microsoft HoloLens and evaluate its performance via both extensive controlled experiments and a user study with 30+ participants. DeepMix not only improves detection accuracy by 9.1-37.3% but also reduces end-to-end latency by 2.68-9.15⇥, compared to the baseline that uses existing 3D object detection models. CCS CONCEPTS• Human-centered computing ! Ubiquitous and mobile computing systems and tools; Systems and tools for interaction design.

show abstract