Segmentation Based Features for Wide-Baseline Multi-view Reconstruction

Abstract

A common problem in wide-baseline stereo is the sparse and non-uniform distribution of correspondences when using conventional detectors such as SIFT, SURF, FAST and MSER. In this paper we introduce a novel segmentation based feature detector SFD that produces an increased number of 'good' features for accurate wide-baseline reconstruction. Each image is segmented into regions by over-segmentation and feature points are detected at the intersection of the boundaries for three or more regions. Segmentation-based feature detection locates features at local maxima giving a relatively large number of feature points which are consistently detected across wide-baseline views and accurately localised. A comprehensive comparative performance evaluation with previous feature detection approaches demonstrates that: SFD produces a large number of features with increased scene coverage; detected features are consistent across wide-baseline views for images of a variety of indoor and outdoor scenes; and the number of wide-baseline matches is increased by an order of magnitude compared to alternative detector-descriptor combinations. Sparse scene reconstruction from multiple wide-baseline stereo views using the SFD feature detector demonstrates at least a factor six increase in the number of reconstructed points with reduced error distribution compared to SIFT when evaluated against ground-truth and similar computational cost to SURF/FAST.

Semantically Coherent Co-Segmentation and Reconstruction of Dynamic Scenes

“…As a result FCNs produce segmentations with poorly localized object boundaries as illustrated in Figure 3(b). Initial Semantic Reconstruction: Sparse feature-based reconstruction of the scene is performed using SFD features [44] and SIFT descriptor [39] with the constraint that each 3D feature should be visible in 3 or more camera views for robustness [26]. The resulting point-cloud is clustered in 3D [50].…”

Temporally Coherent 4D Reconstruction of Complex Dynamic Scenes

“…Extrinsic parameters are calibrated automatically [21,23] using sparse wide-baseline feature matching. Segmentation-based feature detection (SFD) [33] is used to Figure 2. Temporally consistent scene reconstruction framework obtain a relatively large number of sparse features suitable for wide-baseline matching which are distributed throughout the scene including on dynamic objects such as people.…”

4D Match Trees for Non-rigid Surface Alignment

“…Segmentation-based Feature Detection: Several feature detection and matching approaches previously used in wide-baseline matching of rigid scenes have been evaluated for wide-timeframe matching between images of non-rigid shape. Figure 2 and Table 1 present results for SIFT [37], FAST [38] and SFD [33] feature detection. This comparison shows that segmentation-based feature detector (SFD) [33] gives a relatively high number of correct matches.…”

“…Figure 2 and Table 1 present results for SIFT [37], FAST [38] and SFD [33] feature detection. This comparison shows that segmentation-based feature detector (SFD) [33] gives a relatively high number of correct matches. SFD detects keypoints at the triple points between segmented regions which correspond to local maxima of the image gradient.…”

“…The first step is to estimate sparse wide-timeframe feature correspondence. Robust feature matching between frames is achieved using a robust segmentation-based feature detector (SFD) previously proposed for wide-baseline stereo correspondence [33]. The 4D Match Tree is constructed as the minimum spanning tree based on the surface overlap and non-rigid shape similarity between pairs of frames estimated from the sparse feature correspondence.…”