A real-time reconstructed 3D environment augmented with virtual objects rendered with correct occlusion

Abstract

Fig. 1: Screenshots of the left view showing correct occlusion of the user's hands and the virtual object (a 3D textured crate). The AR marker cannot be seen because it is directly behind the virtual object.Abstract-In this work we present a novel framework for the real-time interaction with 3D models in augmented virtual reality. Our framework incorporates view-dependent stereoscopic rendering of the reconstructed environment together with user's hands and a virtual object, and high-precision gesture recognition to manipulate it. Proposed setup consists of a Creative RGB-D camera, Oculus Rift VR head mounted display (HMD), Leap Motion hands and fingers tracker and an AR marker. The system is capable of augmenting the user's hands in relation to their point of view (POV) using the depth sensor mounted on the HMD, and allows manipulation of the environment through the Leap Motion sensor. The AR marker is used to determine the location of the Leap Motion sensor to help with consolidation of transformations between the Oculus and the Leap Motion sensor. Combined with accurate information from the Oculus HMD, the system is able to track the user's head and fingers, with 6-DOF, to provide a spatially accurate augmentation of the user's virtual hands. Such an approach allows us to achieve high level of user immersion since the augmented objects occlude the user's hands properly; something which is not possible with conventional AR. We hypothesize that users of our system will be able to perform better object manipulation tasks in this particular augmented VR setup as compared to virtual reality (VR) where user's hands are not visible, or if visible, always occlude virtual objects.

Implementing Native Support for Oculus and Leap Motion in a Commercial Engineering Visualization and Analysis Platform

“…While using the lenses to increase the devices field of view creates a more immersive experience for the user, the distortion created in the image is undesired. This pincushion distortion, shown in Figure 7, is rectified in the Mockup rendering engine by applying what is know as a barrel distortion, shown in Figure 8 [50]. This distortion correction is applied to the rendered scene using OpenGL shaders within the standard Mockup graphics pipeline.…”

Using CNNs For Users Segmentation In Video See-Through Augmented Virtuality

“…But except for a couple of high-end complex systems [10], they also usually suffer from the improving but still rather poor quality of the depth maps provided by current hardware [11]. In addition, most reconstruction-based systems rely on said outside-in setup [3], [9] (as opposed to head-mounted, inside-out, systems [12]) that may lead to occlusions preventing a whole reconstruction of the users. Other approaches provide depth perception without reconstruction steps by presenting synchronized stereoscopic video pairs to the user's eyes.…”

Enhanced Positioning Algorithm Using a Single Image in an LCD-Camera System by Mesh Elements’ Recalculation and Angle Error Orientation

“…Several similar methods for calculating the position and orientation of a camera in space using a single image have been described and presented [22,25,26]. Nevertheless, pose estimation and marker detection are widely used tasks for many other technological applications such as autonomous robots [27][28][29], unmanned vehicles [30][31][32][33][34][35][36][37], and virtual assistants [38][39][40][41], among others.…”