Locally Consistent ToF and Stereo Data Fusion

Abstract

Abstract. Depth estimation for dynamic scenes is a challenging and relevant problem in computer vision. Although this problem can be tackled by means of ToF cameras or stereo vision systems, each of the two systems alone has its own limitations. In this paper a framework for the fusion of 3D data produced by a ToF camera and a stereo vision system is proposed. Initially, depth data acquired by the ToF camera are up-sampled to the spatial resolution of the stereo vision images by a novel up-sampling algorithm based on image segmentation and bilateral filtering. In parallel a dense disparity field is obtained by a stereo vision algorithm. Finally, the up-sampled ToF depth data and the disparity field provided by stereo vision are synergically fused by enforcing the local consistency of depth data. The depth information obtained with the proposed framework is characterized by the high resolution of the stereo vision system and by an improved accuracy with respect to the one produced by both subsystems. Experimental results clearly show how the proposed method is able to outperform the compared fusion algorithms.

A Survey on Time-of-Flight Stereo Fusion

“…Additionally, based on the requirements discussed in Section 2 we will propose some new experiments and performance measures that we believe will help in a better understanding of the fusion system. [33,40] contain simple synthetic scenes as well as measured tabletop scenes containing a varied amount of textured objects. The reference data was obtained using space time stereo [41] and aligned with ToF data from a MESA SR4000.…”

“…They are often based on a line search that is guided by the ToF data. Global methods, [5,31,35,27,44,26,32,51,52,40,45] add the ToF information as an additional data term in a global energy functional is then jointly optimized. While the depth maps obtained are smoother due to the usage of prior information/regularizers, this is at the cost of additional computational resources.…”

“…While [31,35,27,44,45] employ different graphical models for inference, [51,32] formulate the problem in a variational framework. The last sub-group of the global methods [5,26,52,40] contains those which use other non-local optimization strategies. After a discussion of commonalities in each group, we will proceed to describe each method in detail.…”

“…Often the Middlebury ground truth dataset [16] is used [40,42] and the ToF view is synthesized from the ground truth data. Though an interesting way to compare the results, the naive implementation currently used is to just downsample the GT depth and add some noise to the obtained depth map.…”

“…Mean squared error, median absolute error. In fusion literature [44,40] often the mean squared error is reported instead of accuracy and precision. For real valued functions this corresponds to the sum of variance and squared bias.…”