Motion-Based Object Segmentation Based on Dense RGB-D Scene Flow

Abstract

Given two consecutive RGB-D images, we propose a model that estimates a dense 3D motion field, also known as scene flow. We take advantage of the fact that in robot manipulation scenarios, scenes often consist of a set of rigidly moving objects. Our model jointly estimates (i) the segmentation of the scene into an unknown but finite number of objects, (ii) the motion trajectories of these objects and (iii) the object scene flow. We employ an hourglass, deep neural network architecture. In the encoding stage, the RGB and depth images undergo spatial compression and correlation. In the decoding stage, the model outputs three images containing a per-pixel estimate of the corresponding object center as well as object translation and rotation. This forms the basis for inferring the object segmentation and final object scene flow. To evaluate our model, we generated a new and challenging, large-scale, synthetic dataset that is specifically targeted at robotic manipulation: It contains a large number of scenes with a very diverse set of simultaneously moving 3D objects and is recorded with a simulated, static RGB-D camera. In quantitative experiments, we show that we outperform state-of-the-art scene flow and motion-segmentation methods on this data set. In qualitative experiments, we show how our learned model transfers to challenging real-world scenes, visually generating better results than existing methods.

Deep part induction from articulated object pairs

“…The quantitative comparison is shown in Table 5. Our flow field and segmentation estimation outperforms [Shao et al 2018] by a large margin. We visualize the prediction results in Figure 10.…”

“…Parts can also be extracted from 3D point flows through direct clustering on point trajectories [Pillai et al 2014;Tzionas and Gall 2016a]. More similarly to our approach, the concurrent learning method by Shao et al [2018] trains a joint flow estimation and segmentation network for motion-based object detection in a scene. However, their approach mainly relies on RGB color to compute flow, and cannot handle complex structures or large articulation differences, as discussed in our results section.…”

“…We compare here our method with the concurrent learning method by Shao et al [2018]. Their method trains a joint flow estimation and segmentation network for motion-based object segmentation.…”

“…Note that our approach only uses the point cloud information, while their approach exploits both the rendered images as well as the partial point clouds. We also rendered our training data into RGB-D pairs, and asked the authors of [Shao et al 2018] to re-train their network on our training set so that they can handle the CAD rendered images. The quantitative comparison is shown in Table 5.…”

Object Discovery in Videos as Foreground Motion Clustering

“…Other works that build off this idea include formulating trajectory clustering as a multi-cut problem [23,24,25] or as a density peaks clustering [46], and detecting discontinuities in the trajectory spectral embedding [15]. More recent approaches include using occlusion relations to produce layered segmentations [43], combining piecewise rigid motions with pre-trained CNNs to merge the rigid motions into objects [7], and jointly estimating scene flow and motion segmentations [39]. We use pixel trajectories in a recurrent neural network to learn trajectory embeddings for motion clustering.…”