Blocks World of Touch: Exploiting the Advantages of All-Around Finger Sensing in Robot Grasping

Abstract

Tactile sensing is an essential capability for a robot to perform manipulation tasks in cluttered environments. While larger areas can be assessed instantly with cameras, Lidars, and other remote sensors, tactile sensors can reduce their measurement uncertainties and gain information of the physical interactions between the objects and the robot end-effector that is not accessible via remote sensors. In this paper, we introduce the novel tactile sensor GelTip that has the shape of a finger and can sense contacts on any location of its surface. This contrasts to other camera-based tactile sensors that either only have a flat sensing surface, or a compliant tip of a limited sensing area, and our proposed GelTip sensor is able to detect contacts from all the directions, like a human finger. The sensor uses a camera located at its base to track the deformations of the opaque elastomer that covers its hollow, rigid, and transparent body. Because of this design, a gripper equipped with GelTip sensors is capable of simultaneously monitoring contacts happening inside and outside its grasp closure. Our extensive experiments show that the GelTip sensor can effectively localize these contacts at different locations of the finger body, with a small localization error of approximately 5 mm on average, and under 1 mm in the best cases. Furthermore, our experiments in a Blocks World environment demonstrate the advantages, and possibly a necessity, of leveraging all-around touch sensing in manipulation tasks. In particular, the experiments show that the contacts at different moments of the reach-to-grasp movements can be sensed using our novel GelTip sensor.

Generation of GelSight Tactile Images for Sim2Real Learning

“…As such, the assumptions in this work, i.e., a flat elastomer, directional light sources and no shadows, are sufficiently valid for sensors that are constructed following these working principles, such as the ones proposed in [7], [28]. More recently, other sensors have relaxed some of these constraints in favour of finger-shaped surfaces [8], increased measuring areas [26], or domed elastomers for increased contact [9]. As such, for each sensor different generalizations would have to be performed, even though the main pipeline, i.e., elastomer deformation modeling followed by illumination rendering, should still apply.…”

“…At this point, GelSight sensors are an entire family with multiple variants, and this work represents the first attempt of simulating a GelSight for Sim2Real learning applications. Future works should seek to extend the proposed method for other GelSight-like sensors, such as the GelTip [8], [32].…”

“…In contrast, GelSight optical tactile sensors [7], [8] are able to attain high resolution tactile images, thanks to exploiting the full raw images of the elastomer deformation captured by the embedded camera. It has been widely applied to various perception and manipulation tasks, e.g., geometry and slip measurement [9], localisation [7], texture recognition [10], [11], [12], and tactile servoing [13].…”

GelTip Tactile Sensor for Dexterous Manipulation in Clutter

“…With their compact design, the GelTip [18] and other GelSight [31,[39][40][41]46] sensors are candidate sensors to be mounted on robotic grippers. Recently, custom grippers built using the GelSight working principle have also been proposed A single set of white LEDs is used.…”

“…In this chapter, we will first review existing optical tactile sensors in Section 1.2, and then we'll look in detail into one example of such image-based tactile sensors, i.e., the GelTip [18,19], in Section 1.3. The GelTip is shaped as a finger, and thus it can be installed on traditional and off-the-shelf grippers to replace its fingers, and enable contacts to be sensed inside and outside the grasp closure that are shown in Figure 1.1.…”