More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch

Abstract

For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic grasping work, however, has been based only on visual input, and thus cannot easily benefit from feedback after initiating contact. In this paper, we investigate how a robot can learn to use tactile information to iteratively and efficiently adjust its grasp. To this end, we propose an end-to-end action-conditional model that learns regrasping policies from raw visuo-tactile data. This model -a deep, multimodal convolutional network -predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions. Our approach requires neither calibration of the tactile sensors, nor any analytical modeling of contact forces, thus reducing the engineering effort required to obtain efficient grasping policies. We train our model with data from about 6,450 grasping trials on a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger. Across extensive experiments, our approach outperforms a variety of baselines at (i) estimating grasp adjustment outcomes, (ii) selecting efficient grasp adjustments for quick grasping, and (iii) reducing the amount of force applied at the fingers, while maintaining competitive performance. Finally, we study the choices made by our model and show that it has successfully acquired useful and interpretable grasping behaviors.

Review of Deep Learning Methods in Robotic Grasp Detection

“…They reported a success rate of 85.92% with a deep network and 84.5% with an SVM. A similar approach was followed by Calandra et al [40] in their work on using tactile sensing in robotic grasp detection.…”

“…This representation described a grasp in a 2D image plane. This representation was improved by Calandra et al to include the 3D depth information by adding the z coordinates to the representation, resulting in a grasp representation, G z = (x, y, z, θ) [40,41]. The G z grasp representation was also used by Murali et al [42] in their approach to detect robotic grasps through the use of tactile feedback and visual sensing.…”

“…They have collected data for 52 different objects. Calandra et al [40] collected data from 9269 grasp trials for 106 unique objects. Pinto et al [37] stated how time consuming it was to collect data for robotic grasping and proposed a novel approach inspired from reinforcement learning.…”

Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

“…Even fewer approaches exploit the complementary nature of vision and touch. Some of them extend their previous work on grasp stability estimation [5,13]. Others perform full manipulation tasks based on multiple input modalities [1,20,31] but require a pre-specified manipulation graph [31], demonstrate only on one task [20,31], or require human demonstration and object CAD models [1].…”

“…[43,61] fuse vision and range sensing and [61] add language labels. While many of these multimodal approaches are trained through a classification objective [5,13,26,70], in this paper we are interested in multimodal representation learning for control. Figure 2: Neural network architecture for multimodal representation learning with self-supervision.…”