Learning to Grasp Without Seeing

Abstract

Can a robot grasp an unknown object without seeing it? In this paper, we present a tactile-sensing based approach to this challenging problem of grasping novel objects without prior knowledge of their location or physical properties. Our key idea is to combine touch based object localization with tactile based regrasping. To train our learning models, we created a large-scale grasping dataset, including more than 30K RGB frames and over 2.8 million tactile samples from 7800 grasp interactions of 52 objects. To learn a representation of tactile signals, we propose an unsupervised auto-encoding scheme, which shows a significant improvement of 4-9% over prior methods on a variety of tactile perception tasks. Our system consists of two steps. First, our touch localization model sequentially "touch-scans" the workspace and uses a particle filter to aggregate beliefs from multiple hits of the target. It outputs an estimate of the object's location, from which an initial grasp is established. Next, our re-grasping model learns to progressively improve grasps with tactile feedback based on the learned features. This network learns to estimate grasp stability and predict adjustment for the next grasp. Re-grasping thus is performed iteratively until our model identifies a stable grasp. Finally, we demonstrate extensive experimental results on grasping a large set of novel objects using tactile sensing alone. Furthermore, when applied on top of a vision-based policy, our re-grasping model significantly boosts the overall accuracy by 10.6%. We believe this is the first attempt at learning to grasp with only tactile sensing and without any prior object knowledge. For supplementary video and dataset see: cs.cmu.edu/GraspingWithoutSeeing.

More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch

“…An indepth exploration of the tradeoffs between data-driven and analytic approaches would an interesting future topic of study. Another concurrent work [32], explores grasping with a 3axis force sensor, but reports comparatively low success rates, focusing instead on tactile localization without vision. Our method uses rich touch sensing that is aware of texture and surface shape, simultaneously incorporates multiple modalities, and can flexibly accommodate additional constraints, such as minimum-force grasps.…”

Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching

“…More recently, researchers have also proposed other, more strongly supervised techniques for inferring object properties from touch. For example [8] proposed estimating the hardness of an object using a convolutional network, while [9], [10], [11] estimated material and textural properties. However, to our knowledge, none of these prior works have demonstrated that object instances (rather than just material properties) can be recognized entirely by touch and matched to corresponding visual observations.…”

“…Aside from recognition and perception, tactile information has also been extensively utilized for directly performing robotic manipulation skills, especially grasping. For example, [12], [13], [14], [10] predicted how suitable a given gripper configuration was for grasping. We take inspiration from these approaches, and use the tactile exploration strategy of [12], whereby the robot "feels" a random location of an object using a two-fingered gripper equipped with two GelSight [15], [16] touch sensors.…”

Swoosh! Rattle! Thump! - Actions that Sound

“…Vision is used to infer the geometric shape [24], track objects [40], infer object categories [26] and even direct control [27]. In recent years, the sense of touch has also received increasing attention for recognition [37] and feedback control [30]. But what about sound?…”