Reduction of contradictory partial occlusion in mixed reality by using characteristics of transparency perception

Abstract

One of the challenges in mixed reality (MR) applications is handling contradictory occlusions between real and virtual objects. The previous studies have tried to solve the occlusion problem by extracting the foreground region from the real image. However, real-time occlusion handling is still difficult since it takes too much computational cost to precisely segment foreground regions in a complex scene. In this study, therefore, we proposed an alternative solution to the occlusion problem that does not require precise foreground-background segmentation. In our method, a virtual object is blended with a real scene so that the virtual object can be perceived as being behind the foreground region. For this purpose, we first investigated characteristics of human transparency perception in a psychophysical experiment. Then we made a blending algorithm applicable to real scenes based on the results of the experiment.

Handling occlusions in augmented reality based on 3D reconstruction method

“…Hence, Tian et al [12] proposed an automatic occlusion handling method, extracting the contour of the occluding real object automatically through calculating the disparity map of the real scene in the first frame, but the occlusion handling result is largely influenced by the automatic contour exaction result in the first frame. To reduce contradictory occlusions, Fukiage et al [13] took advantage of characteristics of human transparency perception without precise foreground-background segmentation, which is proved to be robust and real time even though there are complicated foreground objects in the scene. Sanches et al [14] segmented the real element in real-time and then performed OpenGL frame buffer operations to recover the pixels belonging to virtual objects to handle mutual occlusion, but there must be only one real object moving and many virtual objects based on fiducial markers.…”

Occlusion handling using semantic segmentation and visibility-based rendering for mixed reality

“…By combining the semantic segments and the probability map, we overlay the CG object onto the real scene by adapting a visibilitybased rendering method rst proposed in [4]. In [4], a blending method, which uses visibility predictor based on human vision system, was used to predict the visibility (or transparency) level of the CG object. e method has an advantage over alpha blending methods ( Figure 2) because it does not require accurate estimation of complex boundaries.…”

“…Instead of using a xed visibility level for all objects, as in [4], we use our proposed semantic classes to choose the amount of visibility for di erent type of objects. is allows us to control the appearance of the rendered object based on the type of the scene.…”

“…In Section 4 we introduce our visibility-based blending method which uses semantic classsi cation and the foreground probability map. In Section 6, we show our results and comparison among our method, alpha blending, and transparency blending [4] methods. Finally, we conclude this work in Section 7.…”

Visibility-based blending for real-time applications

“…For example, in portable augmented reality systems, rendering virtual information in 100% opacity can be dangerous because obstacles in the real world are often occluded. Virtual objects may also be rendered transparently for the purpose of X-ray visualizations [6,26]. In showing virtual objects in optical see-through systems, or structured augmented reality systems, virtual information is usually perceived half-transparently.…”