N-tuple Color Segmentation for Multi-view Silhouette Extraction

Abstract

We present a new method to extract multiple segmentations of an object viewed by multiple cameras, given only the camera calibration. We introduce the n-tuple color model to express interview consistency when inferring in each view the foreground and background color models permitting the final segmentation. A color n-tuple is a set of pixel colors associated to the n projections of a 3D point. The first goal is set as finding the MAP estimate of background/foreground color models based on an arbitrary sample set of such n-tuples, such that samples are consistently classified, in a soft way, as "empty" if they project in the background of at least one view, or "occupied" if they project to foreground pixels in all views. An Expectation Maximization framework is then used to alternate between color models and soft classifications. In a final step, all views are segmented based on their attached color models. The approach is significantly simpler and faster than previous multi-view segmentation methods, while providing results of equivalent or better quality.

3D reconstruction for featureless scenes with curvature hints

“…Additionally, we show in Fig. 2 a comparison with an automatic state of the art multi-view image segmentation method, i.e., the approach presented in [12], on a dataset where the object of interest, the car, is not fully visible in all of the images. It can be seen how, although both the algorithms produce acceptable results, our approach is able to correctly classify the car's pixel even behind vegetation.…”

“…It can be seen how, although both the algorithms produce acceptable results, our approach is able to correctly classify the car's pixel even behind vegetation. Moreover, the technique [12] applied to the Museum dataset (the one used in Fig. 1) cannot produce any usable result due to the fact that the object of interest (the tree) is not in the center of the images.…”

“…In a subsequent work [9] by the same authors, the voxel grid was replaced by adding stereo correspondences and epipolar constraints to directly connect pixels from different images. Djelouah et al [12] use a set of sample points uniformly distributed in the volume (under a similar hypothesis as in [10]) and consider the tuple of pixels defined by the projection of the sample point on the images. The key idea is that if all the elements of a tuple are classified as foreground, then the point belongs to the object's surface.…”

Wide Baseline Multi-view Video Matting Using a Hybrid Markov Random Field

“…The MRF is solved once per frame to extract the binary mattes for all views simultaneously. Our solution of all views in a single step contrasts with existing methods that either propagate labels between pairs of independently solved views [5], or build foreground appearance models across all views but then independently solve the segmentation for each view [6]. Additionally, we contribute a propagation strategy in which soft labelling constraints are carried forward in time to influence the MRFs defined over subsequent frames, so enhancing the coherence of the resulting matte sequence.…”

“…Weaker geometric information is incorporated by Sarim et al who propagate trimaps between views by matching along the epipolar line [5]. Recently Djelouah et al [6] proposed a geometry-free approach, building foreground and background appearance models simultaneously across views using expectation maximisation. However to perform the segmentation itself a set of independent graph-cuts (via GrabCut [19]) are used to extract a matte from each view in isolation using those models.…”