Estimating scene flow using an interconnected patch surface model with belief-propagation inference

Abstract

This article presents a novel method for estimating the dense three-dimensional motion of a scene from multiple cameras. Our method employs an interconnected patch model of the scene surfaces. The interconnected nature of the model means that we can incorporate prior knowledge about neighbouring scene motions through the use of a Markov Random Field, whilst the patchbased nature of the model allows the use of efficient techniques for estimating the local motion at each patch. An important aspect of our work is that the method takes account of the fact that local surface texture strongly dictates the accuracy of the motion that can be estimated at each patch. Even with simple squared-error cost functions, it produces results that are either equivalent to or better than results from a method based upon a state-of-the-art optical flow technique, which uses well-developed robust cost functions and energy minimisation techniques.

SS-SF: Piecewise 3D Scene Flow Estimation With Semantic Segmentation

“…Although the variational computation framework is capable of producing dense scene flow field, the global smoothing assumption of the classical model may be sensitive to motion discontinuities and occlusions. Popham et al [48] employed an interconnected patch model to modify the classical scene flow model, which estimates the accurate scene flow at each region. Bleyer et al [49] recommended a soft constraint by using object-level color models to address the issue of motion occlusion.…”

Scene Flow Estimation: A Survey

“…To overcome this problem, different scene representations have been utilized. For example, Popham introduced a patch-based method [32]. The motion of each patch doesn't only rely on the texture information, but utilizes the motion from neighbor patches.…”

“…Mesh is essentially sort of point cloud representation as the vertex can be viewed as a point in the three-dimensional point cloud. Scene flow estimation methods with a mesh representation [61,62,63,42,64,65,66,67,32] are only under a multi-view setting and the geometry estimation is given simultaneously. The motion of vertice is solved with a point cloud methods, while the rest part are solved by interpolating along the meshes.…”

“…Early paper viewed patch as a surface element(surfel) under a multi-view setting [68,69]. A few binocular-based scene flow methods utilized patches to fit the surface of the scene [43,70,71,32,72,73,31] on account that this kind of patch-based methods are common in stereo matching field. In addition, Hornacek uniquely exploited a pair of RGB-D data to seek patch correspondences in the 3D world space and leads to dense body motion field including both translation and rotation [74].…”

“…Similarly, Pons presented a common variational framework with local similarity criteria constraint [75,76]. Henceforth, different scene representations were introduced to describe the surface [68,61,62,63,42,70,65,32]. Diverse multi-frame tracking methods mentioned for sparse estimation are utilized as well to build the temporal coherence [36,69,64,67].…”