A Variational Method for Scene Flow Estimation from Stereo Sequences

Abstract

International audienceThis paper presents a method for scene flow estimation from a calibrated stereo image sequence. The scene flow contains the 3-D displacement field of scene points, so that the 2-D optical flow can be seen as a projection of the scene flow onto the images. We propose to recover the scene flow by coupling the optical flow estimation in both cameras with dense stereo matching between the images, thus reducing the number of unknowns per image point. The use of a variational framework allows us to properly handle discontinuities in the observed surfaces and in the 3-D displacement field. Moreover our approach handles occlusions both for the optical flow and the stereo. We obtain a partial differential equations system coupling both the optical flow and the stereo, which is numerically solved using an original multi-resolution algorithm. Whereas previous variational methods were estimating the 3-D reconstruction at time $t$ and the scene flow separately, our method jointly estimates both in a single optimization. We present numerical results on synthetic data with ground truth information, and we also compare the accuracy of the scene flow projected in one camera with a state-of-the-art single-camera optical flow computation method. Results are also presented on a real stereo sequence with large motion and stereo discontinuities. Source code and sample data are available for the evaluation of the algorithm

Disparity Estimation in Stereo Sequences using Scene Flow

“…The best results in terms of accuracy were obtained by Brox et al [13]. The global energy is only linearized inside the minimization algorithm after warping the image at time t+1 on to the image at time t. [14] further extends optical flow into scene flow estimation. The method computes scene flow by joint estimation of the disparity maps and the motion field from a calibrated stereoscopic image sequence within a unified variational framework.…”

“…This paper adopts the approach in [14,15] to estimate the 3D motion, but for a different purpose. A prior probabilistic model is built from the scene flow estimated and used in stereo estimation at time t+1.…”

“…We use the viewdependent representation of the scene flow similar to the work by Huguet [14]. The optical flow and disparity maps are jointly estimated from the two rectified image sequences.…”

“…This process is repeated until the full resolution is reached. In our implementation, we use 0.8 for the down sampling factor, and as in [14], 0.05 is used as the stopping condition for the inner fixed point iterations and 0.01 for the outer fixed point iterations.…”

“…The approach takes the advantages of an over-segmentation based stereo algorithm and integrates it into a pixel based algorithm, which leads to good performance on a single pair of stereo images. Furthermore, we model the temporal consistency of disparity maps using scene flow [14] (also known as disparity flow), which captures the dense 3D motion in the scene from a given view. The scene flow is built into the stereo models for the next frame as a soft constraint, which helps to resolve stereo ambiguity from the temporal domain, while at the same time reducing the error propagation in disparity maps.…”