Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching

Abstract

This paper presents a robotic pick-and-place system that is capable of grasping and recognizing both known and novel objects in cluttered environments. The key new feature of the system is that it handles a wide range of object categories without needing any task-specific training data for novel objects. To achieve this, it first uses an object-agnostic grasping framework to map from visual observations to actions: inferring dense pixel-wise probability maps of the affordances for four different grasping primitive actions. It then executes the action with the highest affordance and recognizes picked objects with a cross-domain image classification framework that matches observed images to product images. Since product images are readily available for a wide range of objects (e.g., from the web), the system works out-of-the-box for novel objects without requiring any additional data collection or re-training. Exhaustive experimental results demonstrate that our multiaffordance grasping achieves high success rates for a wide variety of objects in clutter, and our recognition algorithm achieves high accuracy for both known and novel grasped objects. The approach was part of the MIT-Princeton Team system that took 1st place in the stowing task at the 2017 Amazon Robotics Challenge. All code, datasets, and pre-trained models are available online at

Robotic grasp detection using deep convolutional neural networks

“…Also, the availability of affordable RGB-D sensors enabled the use of deep learning techniques to learn the features of objects directly from image data. Recent experimentations on Convolutional neural network [2], [17], [18] have demonstrated that they can be used to efficiently compute stable grasps, Pinto et al [3] used an architecture similar to AlexNet to depict that by increasing the size of the data, their CNN was able to generalize better to new data. Varley et al [19] propose an interesting approach to grasp planning through shape completion where a 3D CNN was used to train the network on 3D prototype of objects in their own dataset captured from various viewpoints.…”

Joint Learning of Instance and Semantic Segmentation for Robotic Pick-and-Place with Heavy Occlusions in Clutter

“…For semantic occlusion segmentation, we extend previous work of semantic (visible) segmentation by fully convolutional networks (FCN) proposed in [1]. All layers are composed of convolutional or pooling layers, which keep the geometry of image, so FCN is known as effective and widely used for pixel-wise score regression tasks: depth prediction [24,25], grasp affordance [12,26,27], optical flow [28], and instance masks [4,6].…”

A review on object pose recovery: From 3D bounding box detectors to full 6D pose estimators

“…One prominent field is robotic manipulation, in which industrial parts are grasped and are placed to a desired location [62], [56], [57], [60], [91], [202], [200]. Amazon Picking Challenge (APC) [1] is an important example demonstrating how object detection and 6D pose estimation, when successfully performed, improves the autonomy of the manipulation, regarding the automated handling of parts by robots [55], [58], [59]. Household robotics is another field where the ability of recognizing objects and accurately estimating their poses is a key element [97], [98], [99].…”