Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching

Abstract

Much of the literature on robotic perception focuses on the visual modality. Vision provides a global observation of a scene, making it broadly useful. However, in the domain of robotic manipulation, vision alone can sometimes prove inadequate: in the presence of occlusions or poor lighting, visual object identification might be difficult. The sense of touch can provide robots with an alternative mechanism for recognizing objects. In this paper, we study the problem of touch-based instance recognition. We propose a novel framing of the problem as multi-modal recognition: the goal of our system is to recognize, given a visual and tactile observation, whether or not these observations correspond to the same object. To our knowledge, our work is the first to address this type of multi-modal instance recognition problem on such a large-scale with our analysis spanning 98 different objects. We employ a robot equipped with two GelSight touch sensors, one on each finger, and a self-supervised, autonomous data collection procedure to collect a dataset of tactile observations and images. Our experimental results show that it is possible to accurately recognize object instances by touch alone, including instances of novel objects that were never seen during training. Our learned model outperforms other methods on this complex task, including that of human volunteers.

Tactile Image Sensors Employing Camera: A Review

“…When the object makes contact with the sensor surface, the elastomer surface deforms, and a shading image reflecting these deformations appears in the image obtained from the camera. This method has been used in GelSight, which is currently the most popular tactile image sensor [32,36,38,40,42,76,77,78,79,80], and some other sensors [37,41,43].…”

Visuotactile Sensors With Emphasis on GelSight Sensor: A Review

“…According to Yuan et al [71], the success of computer vision has been greatly accelerated through the use of deep learning. CNN was used in many GelSight studies related to object recognition and classification as well as cross-modal analysis as reported in [54], [56], [70]- [79].…”

PUT-Hand—Hybrid Industrial and Biomimetic Gripper for Elastic Object Manipulation

“…Equipping a gripper with tactile sensors can further improve its manipulation capabilities in robotised setups by providing feedback to the robot controller [51], estimating the contact force and actively controlling the reaction forces to stabilise the grasp [52], or as a source of object identification [53,54]. Tactile information can also be used to determine physical properties of an elastic object or to determine the state of manipulated object.…”