Passthrough+

Abstract

We present an end-to-end system for real-time environment capture, 3D reconstruction, and stereoscopic view synthesis on a mobile VR headset. Our solution allows the user to use the cameras on their VR headset as their eyes to see and interact with the real world while still wearing their headset, a feature often referred to as Passthrough. The central challenge when building such a system is the choice and implementation of algorithms under the strict compute, power, and performance constraints imposed by the target user experience and mobile platform. A key contribution of this paper is a complete description of a corresponding system that performs temporally stable passthrough rendering at 72 Hz with only 200 mW power consumption on a mobile Snapdragon 835 platform. Our algorithmic contributions for enabling this performance include the computation of a coarse 3D scene proxy on the embedded video encoding hardware, followed by a depth densification and filtering step, and finally stereoscopic texturing and spatio-temporal up-sampling. We provide a detailed discussion and evaluation of the challenges we encountered, as well as algorithm and performance trade-offs in terms of compute and resulting passthrough quality.;AB@The described system is available to users as the Passthrough+ feature on Oculus Quest. We believe that by publishing the underlying system and methods, we provide valuable insights to the community on how to design and implement real-time environment sensing and rendering on heavily resource constrained hardware.

Towards a Framework for Validating XR Prototyping for Performance Evaluations of Simulated User Experiences

“…In our case, participants remarked on issues of depth perception, likely due to the offset between the users' eyes and the VST cameras. A follow-up study may explore, if eye-perspective rendering provides a solution for this type of technology [17,23,24]. Overall, as a validation study can likely contain multiple confounding factors making absolute validity potentially infeasible to prove, studies may focus on demonstrating relative validity of results.…”

“…Unsurprisingly, wearing an HMD influences users and their performance in XRP validation studies. As depth perception was identified as a major issue by participants, it is likely better to utilize a VST HMD that support rendering the camera viewpoint from the user's eye perspective [17,23,24], which may improve depth perception. To navigate this issue, validation studies may also focus on evaluating user experiences that require the same or similar display hardware to make this confound part of the experience design, e.g., when evaluating novel adaptive MR interfaces [18,25] that require wearing HMDs in any case.…”

Comparing Usability of Augmented Reality and Virtual Reality for Creating Virtual Bounding Boxes of Real Objects

“…The Quest 2 provides the Passthrough feature, which uses the device's various cameras and infrared sensors to visualize the user's real-world surroundings [14], as seen in Figure 2. As a result, both of these devices can display virtual objects co-located and visualized in the real world.…”