General Dynamic Scene Reconstruction from Multiple View Video

Abstract

This paper introduces a general approach to dynamic scene reconstruction from multiple moving cameras without prior knowledge or limiting constraints on the scene structure, appearance, or illumination. Existing techniques for dynamic scene reconstruction from multiple wide-baseline camera views primarily focus on accurate reconstruction in controlled environments, where the cameras are fixed and calibrated and background is known. These approaches are not robust for general dynamic scenes captured with sparse moving cameras. Previous approaches for outdoor dynamic scene reconstruction assume prior knowledge of the static background appearance and structure. The primary contributions of this paper are twofold: an automatic method for initial coarse dynamic scene segmentation and reconstruction without prior knowledge of background appearance or structure; and a general robust approach for joint segmentation refinement and dense reconstruction of dynamic scenes from multiple wide-baseline static or moving cameras. Evaluation is performed on a variety of indoor and outdoor scenes with cluttered backgrounds and multiple dynamic non-rigid objects such as people. Comparison with state-of-the-art approaches demonstrates improved accuracy in both multiple view segmentation and dense reconstruction. The proposed approach also eliminates the requirement for prior knowledge of scene structure and appearance.

Learning Dense Wide Baseline Stereo Matching for People

“…Daisy [34] and Normalized Cross Correlation (NCC) [12] uses gradient of local patch's around pixels to compute descriptor or pixel colour distribution of local patch [12] to measure patch correlation for dense wide baseline matching. In previous approaches, computation of a patch similarity measure is used as a photo-metric loss term in the objective function of optimization schema which exploits other priors, such as optical flow, edges or foreground/background segmentation [24,28,15]. In other words, dynamic wide baseline stereo reconstruction has not been considered as an individual solution.…”

“…However, for wide baseline stereo, the research has focused on conventional methods [34,12], and data driven learning based approaches are still an open research question because of the lack of training data. Results from [15,28,24] demonstrate that conventional wide baseline stereo methods have limitation on finding accurate matching for the human body surface. Inspired by the narrow-baseline learning based approaches and need for human specific wide baseline stereo matching, we propose a framework to estimate wide baseline dense stereo matching for people.…”

Segmentation Based Features for Wide-Baseline Multi-view Reconstruction

“…This is then refined for each object through joint optimisation of shape and segmentation using a robust cost function for wide-baseline matching. View-dependent opti- misation of depth is performed with respect to each camera which is robust to errors in camera calibration and initialisation to obtain dense reconstruction [31].…”

“…We initialize the reconstruction and segmentation refinement algorithm [31] using sparse reconstruction obtained from the proposed algorithm. The results are shown in Figure 8.…”

4D Match Trees for Non-rigid Surface Alignment

“…These approaches typically produce an independent 3D scene model at each time instant with partial and erroneous surface reconstruction for moving objects due to occlusion and inherent visual ambiguity [1,2,3,4]. For non-rigid objects, such as people with loose clothing or animals, producing a temporally coherent 4D representation from partial surface reconstructions remains a challenging problem.…”