User-Perspective AR Magic Lens from Gradient-Based IBR and Semi-Dense Stereo

Abstract

We present a new approach to rendering a geometrically-correct user-perspective view for a magic lens interface, based on leveraging the gradients in the real world scene. Our approach couples a recent gradient-domain image-based rendering method with a novel semi-dense stereo matching algorithm. Our stereo algorithm borrows ideas from PatchMatch, and adapts them to semi-dense stereo. This approach is implemented in a prototype device build from off-the-shelf hardware, with no active depth sensing. Despite the limited depth data, we achieve high-quality rendering for the user-perspective magic lens.

A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects

“…In see-through, image-based methods are preferred for seamless real-virtual composition. Homography transformations of images [13,19,35,36,41,42], rendering texture 3D models [6,31,43], rendering voxels [37], and image-based rendering (IBR) [25,26,30,44] are the major approaches. In inpaint, pixels or image patches are synthesized in the ROI to fill in the region with plausible pixels [9,10,21,38].…”

“…Almost all of the existing DR methods allow six degrees of freedom (6DoF) motion for the main camera using positioning sensors [33,34], fiducial markers [22,25,41,49], model-based tracking [13,26], vSLAM [14,23,27,35,44,53], etc. Cosco et al used ARToolKit and ARToolKitPlus markers to track the main camera at pre-observation and run-time for indoor desktop a DR scenario [25,49].…”

“…In order to decrease the inconsistency, facing planes [42], arbitrary plane [35,36], non-planar [43,44], and simplified versions [85,86] of these methods have been developed. It has also been shown that users prefer this display method [87] and the efficiency of the search operation is improved in some cases [88,89].…”

Temporal Upsampling of Depth Maps Using a Hybrid Camera

“…is the index of the pixel obtained by applying the translation of v s,i to the index i [51]. div(s, i) is the divergence of the optical flow.…”

User-Perspective Rendering for Handheld Applications

“…To develop a more general calibration approach, Baričević et al [2] propose a calibration algorithm using the mirror-based method developed by Matsuyama et al [15]. Though applicable on many devices, this calibration method requires measuring the device geometry in advance.…”