Layered RGBD scene flow estimation

Abstract

Our layered approach can handle multiple moving objects and reliably estimate their motion in occlusion regions. Our key observation is that depth provides the depth ordering information, thereby solving a computational bottleneck for previous RGB layered methods (please see Figure 6 for our detected occlusions and the estimated motion by the recent semi-rigid scene flow (SRSF) method [27]). AbstractAs consumer depth sensors become widely available, estimating scene flow from RGBD sequences has received increasing attention. Although the depth information allows the recovery of 3D motion from a single view, it poses new challenges. In particular, depth boundaries are not wellaligned with RGB image edges and therefore not reliable cues to localize 2D motion boundaries. In addition, methods that extend the 2D optical flow formulation to 3D still produce large errors in occlusion regions. To better use depth for occlusion reasoning, we propose a layered RGBD scene flow method that jointly solves for the scene segmentation and the motion. Our key observation is that the noisy depth is sufficient to decide the depth ordering of layers, thereby avoiding a computational bottleneck for RGB layered methods. Furthermore, the depth enables us to estimate a per-layer 3D rigid motion to constrain the motion of each layer. Experimental results on both the Middlebury and real-world sequences demonstrate the effectiveness of the layered approach for RGBD scene flow estimation.

Temporally Consistent Motion Segmentation From RGB‐D Video

“…They do not address segmentation, and their method processes pairs of RGB-D frames separately. Sun et al [SSP15] also address scene flow, but they formulate an energy over several frames in terms of scene segmentation and flow. While they can deal with several moving objects, their segmentation is separating depth layers, not objects.…”

Motion Cooperation: Smooth Piece-wise Rigid Scene Flow from RGB-D Images

“…[20] partitions the scene into depth layers, [24] divides the scene into piecewise planar regions). Therefore, the segmentation-from-motion problem, which can be particularly useful for scene understanding or human-machine interaction, is not trully addressed by these methods.…”

“…Each motion segment is assigned one rigid-body motion, but the approach does not interpolate between the motions of the segments. Recently, Sun et al [20] proposed a probabilistic approach which makes use of a depth-based segmentation to estimate motion between RGB-D images. They regularize the estimation process by retrieving a mean rigid-body motion in each layer.…”

“…For all the sequences, the scene flow is evaluated quantitatively and compared with three state-of-the-art methods: the Primal-Dual flow (PD-Flow) [12], the Semi-Rigid flow (SR-Flow) [16] and the Layered flow [20]. First, the photometric and geometric residuals are computed by warping the intensity and depth images (respectively) according to the estimated flow.…”

Real-time Estimation of Physical Activity Intensity for Daily Living

“…Motion information could also be recovered directly using the optical flow derived from two adjacent RGB images [22] or 4D surface normals [14] and more recently, dense 3D flow [20] from depth images. These approaches, however, often suffer from unaffordable run time, for example with a reported computation time of up to 9 minutes per frames in [20].…”