Learning to Singulate Objects Using a Push Proposal Network

Abstract

Learning to act in unstructured environments, such as cluttered piles of objects, poses a substantial challenge for manipulation robots. We present a novel neural network-based approach that separates unknown objects in clutter by selecting favourable push actions. Our network is trained from data collected through autonomous interaction of a PR2 robot with randomly organized tabletop scenes. The model is designed to propose meaningful push actions based on over-segmented RGB-D images. We evaluate our approach by singulating up to 8 unknown objects in clutter. We demonstrate that our method enables the robot to perform the task with a high success rate and a low number of required push actions. Our results based on real-world experiments show that our network is able to generalize to novel objects of various sizes and shapes, as well as to arbitrary object configurations. Videos of our experiments can be viewed at

Self-supervised Transfer Learning for Instance Segmentation through Physical Interaction

“…All data in ROBOTPUSH is collected autonomously by the robot, which uses a learning-based method for object separation (from own prior work) that effectively isolates cluttered objects using push actions [35]. In a second experiment, we perform a fine-grained evaluation of segmentation performance with respect to each push interaction, which provides insights in segmentation performance for various degrees of clutter.…”

“…In comparison to random pushing this mitigates the problem of moving objects into each other or moving two objects at the same time. One approach for realizing this has been described in our previous work [35]. We employ this method in the approach described in this paper.…”

“…Figure 6 qualitatively shows the improved instance segmentation after each push interaction. The push actions were chosen based on a pre-trained CNN for object separation from own prior work [35]. The results suggest that following an interactive perception strategy substantially improves the segmentation performance at test time and in addition provides the selfsupervised data for training a more accurate network.…”

A Deep Learning Approach to Grasping the Invisible

“…To mitigate uncertainty and collision introduced by clutter, non-prehensile manipulations [10], such as pushing, are investigated in both model-driven approaches [11], [12], [13] and data-driven approaches [1], [14], [15]. With the addition of pushing, push-grasping systems [16], [3] are advanced.…”

“…While some targetoriented manipulations [4], [5] have been applied to the *This work was in part supported by the MnDRIVE Initiative on Robotics, Sensors, and Advanced Manufacturing. 1 The target object is the green cylinder and initially invisible to the robot. We propose to solve the problem by the exploration-singulation-grasping procedure.…”