Real-Time Visibility-Based Fusion of Depth Maps

Abstract

We present a viewpoint-based approach for the quick fusion of multiple stereo depth maps. Our method selects depth estimates for each pixel that minimize violations of visibility constraints and thus remove errors and inconsistencies from the depth maps to produce a consistent surface. We advocate a two-stage process in which the first stage generates potentially noisy, overlapping depth maps from a set of calibrated images and the second stage fuses these depth maps to obtain an integrated surface with higher accuracy, suppressed noise, and reduced redundancy. We show that by dividing the processing into two stages we are able to achieve a very high throughput because we are able to use a computationally cheap stereo algorithm and because this architecture is amenable to hardwareaccelerated (GPU) implementations. A rigorous formulation based on the notion of stability of a depth estimate is presented first. It aims to determine the validity of a depth estimate by rendering multiple depth maps into the reference view as well as rendering the reference depth map into the other views in order to detect occlusions and freespace violations. We also present an approximate alternative formulation that selects and validates only one hypothesis based on confidence. Both formulations enable us to perform video-based reconstruction at up to 25 frames per second. We show results on the Multi-View Stereo Evaluation benchmark datasets and several outdoors video sequences. Extensive quantitative analysis is performed using an accurately surveyed model of a real building as ground truth.

Segmentation and Recognition Using Structure from Motion Point Clouds

“…This motion-covariant feature is naturally dependent on the extent to which objects move, so should help separate buildings from cars, for example. 2 This cue is illustrated in the supplementary video.…”

“…Our algorithm is able to accurately recognize objects and segment video frames without appearance-based descriptors or dense depth estimates obtained using e.g., dense stereo or laser range finders. The structure from motion, or SfM, community [1] has demonstrated the value of ego-motion derived data, and their modeling efforts have even extend to stationary geometry of cities [2]. However, the object recognition opportunities presented by the inferred motion and structure have largely been ignored 1 .…”

A Generative Model for Online Depth Fusion

“…Given a video (and view pose) from a moving camera, we obtain coarse disparity estimates by finding maxima of normalized cross correlation (NCC) scores for 5×5 windows between pairs of frames, similar to [17], and fuse these estimates. 3 We quantitatively evaluate the methods by texture mapping one of our synthetic scenes (with sensor motion) and applying this approach. The results, in figure 7, show TSDF performing significantly better, and in particular the Generative methods suffering from a lack of disparity accuracy ( fig.…”

“…Some model pixel depths in an MRF, determining visibility from those depths [1,2], a general approach that has since been very popular, and even made real-time [3]. Others model voxel occupancies, initially considering visibility by visiting unoccluded regions of space first [4], but more recently using complex 3D MRF formulations [5,6], computing depth as a 3D segmentation between visible and invisible points [5], or computing probabilistic occupancies using long-range ray cliques to model visibility [6].…”

Recovering consistent video depth maps via bundle optimization

“…Gargallo and Sturm [6] proposed to formulate the 3D modeling from images as a Bayesian MAP problem, using multiple depth maps. Recently, Merrell et al [11] proposed a quick depth map fusion method to construct a consistent surface. They employed a weighted blending method based on the visibility constraint and confidences.…”