Efficient 3D Object Segmentation from Densely Sampled Light Fields with Applications to 3D Reconstruction

Abstract

Precise object segmentation in image data is a fundamental problem with various applications, including 3D object reconstruction. We present an efficient algorithm to automatically segment a static foreground object from highly cluttered background in light fields. A key insight and contribution of our article is that a significant increase of the available input data can enable the design of novel, highly efficient approaches. In particular, the central idea of our method is to exploit high spatio-angular sampling on the order of thousands of input frames, for example, captured as a hand-held video, such that new structures are revealed due to the increased coherence in the data. We first show how purely local gradient information contained in slices of such a dense light field can be combined with information about the camera trajectory to make efficient estimates of the foreground and background. These estimates are then propagated to textureless regions using edge-aware filtering in the epipolar volume. Finally, we enforce global consistency in a gathering step to derive a precise object segmentation in both 2D and 3D space, which captures fine geometric details even in very cluttered scenes. The design of each of these steps is motivated by efficiency and scalability, allowing us to handle large, real-world video datasets on a standard desktop computer. We demonstrate how the results of our method can be used for considerably improving the speed and quality of image-based 3D reconstruction algorithms, and we compare our results to state-of-the-art segmentation and multiview stereo methods.

A self-adaptive segmentation method for a point cloud

“…Fig. 8 presents some segmentation results of models that we obtained from [14] and [26]. We note that the algorithm is robust when applied to these models and preserves the connectivity of the point cloud.…”

Light Field Image Processing: An Overview

“…Unlike conventional segmentation approaches, such as co-segmentation and multiview segmentation, views in a light field are much more correlated (i.e., EPI volume), and thus can help in ensuring labeling consistency. However, due to the large number of views in a light field (usually a dozen [157] to a hundred views [158], [159]), segmentation and matting using light field data take abundant computing resources and running time during the energy minimization, which restricts their further promotion. Campbell et al presented multiple-view segmentation approaches using graph-cuts in voxel space [161] and image space [162].…”

“…Wanner et al [157] presented a variational framework for multi-label segmentation on the ray space of 4D light fields. Yücer et al [158], [159] proposed a segmentation approach that uses unstructured 3D light fields captured by a hand-held video camera (see Fig. 16(a)).…”

“…Further examples of applications using light fields. (a) Object segmentation proposed by Yücer et al [158], [159], where the likelihood of edges belonging to the foreground was estimated using the gradients in the 3D light field volume. The left column shows a sub-aperture image and an EPI of the light field, the middle column shows the segmentation results for them, and the right column shows the 3D segmentation result.…”

Improved Sliding Shapes for Instance Segmentation of Amodal 3D Object

“…The core goal of our network is to achieve point-wise amodal 3D instance segmentation [31,32,33] without increasing the running time of existing 3D detection streamline(e.g, Deep Sliding Shapes network [3]). To this end, we extend the Deep Sliding Shapes by two steps.…”