Real-Time 3D Reconstruction for Occlusion-Aware Interactions in Mixed Reality

Abstract

Abstract. In this paper, we present a system for performing real-time occlusion-aware interactions in a mixed reality environment. Our system consists of 16 ceiling-mounted cameras observing an interaction space of size 3.70 m x 3.20 m x 2.20 m. We reconstruct the shape of all objects inside the interaction space using a visual hull method at a frame rate of 30 Hz. Due to the interactive speed of the system, the users can act naturally in the interaction space. In addition, since we reconstruct the shape of every object, the users can use their entire body to interact with the virtual objects. This is a significant advantage over marker-based tracking systems, which require a prior setup and tedious calibration steps for every user who wants to use the system. With our system anybody can just enter the interaction space and start interacting naturally. We illustrate the usefulness of our system through two sample applications. The first application is a real-life version of the well known game Pong. With our system, the player can use his whole body as the pad. The second application is concerned with video compositing. It allows a user to integrate himself as well as virtual objects into a prerecorded sequence while correctly handling occlusions.

Foreground and shadow occlusion handling for outdoor augmented reality

“…Pilet et al proposed a method to re-texture the deformable surface with the presence of complex illumination and occlusion in [20] but outdoor illumination was not taken into account. While Kanbara [9] and Kim [11] applied stereo vision for depth information for which multiple cameras are required, Ladikos et al used a system of 16 ceiling-mounted cameras to let users interact with virtual objects [12]. However, those works rely on accurate stereo vision which is time-consuming.…”

Reduction of contradictory partial occlusion in mixed reality by using characteristics of transparency perception

“…[10] and [11] reconstructed depth information in a real scene to detect occlusion using a stereo vision-based technique. [13], [2], and [8] handled occlusion by constructing the visual hull [14] of objects using multiple cameras. Although some of these studies enabled real-time interaction in an MR scene without contradictory occlusion, the use of the applications was restricted to a specific local space and not applicable to arbitrary outdoor scenes.…”