Sim-to-Real for High-Resolution Optical Tactile Sensing: From Images to Three-Dimensional Contact Force Distributions

Abstract

The images captured by vision-based tactile sensors carry information about high-resolution tactile fields, such as the distribution of the contact forces applied to their soft sensing surface. However, extracting the information encoded in the images is challenging and often addressed with learning-based approaches, which generally require a large amount of training data. This article proposes a strategy to generate tactile images in simulation for a vision-based tactile sensor based on an internal camera that tracks the motion of spherical particles within a soft material. The deformation of the material is simulated in a finite element environment under a diverse set of contact conditions, and spherical particles are projected to a simulated image. Features extracted from the images are mapped to the three-dimensional contact force distribution, with the ground truth also obtained using finite-element simulations, with an artificial neural network that is therefore entirely trained on synthetic data avoiding the need for real-world data collection. The resulting model exhibits high accuracy when evaluated on real-world tactile images, is transferable across multiple tactile sensors without further training, and is suitable for efficient real-time inference.

Simulation, Learning, and Application of Vision-Based Tactile Sensing at Large Scale

“…Tactile Gym was validated against finger-sized TacTip sensors of either hemispherical or rectangular shapes, in which the gap between real and virtual images was mitigated using a trained generative framework. Alternatively, commercial FEM-based simulators (e.g., Abaqus [23], [24]) offer a systematic way to accomplish this challenge by dividing the soft body into many subelements, which are then dynamically analyzed with hyperelastic material models. However, extreme computational costs and inflexibility of the commercial FEM simulators restrict the effective application of these methods in real-time scenarios.…”

“…1) Corotational FEM Approach: The softness of TacLink skin derives from the inherently nonlinear property of soft materials that always pose a critical challenge to mechanical modeling. One can tackle this issue with hyperelastic material models available in off-the-shelf simulation platforms [24], [28]. However, it requires tremendous effort to accurately identify all the necessary parameters through either experimental or numerical processes.…”

Learning Haptic-Based Object Pose Estimation for In-Hand Manipulation Control With Underactuated Robotic Hands

“…Some work have harnessed Finite Element Methods (FEM) in order to generate simulated deformation of the contact pad in tactile sim-to-real [20]. However, the computational complexity of such an approach limits real-time sensing [21].…”

Event Augmentation for Contact Force Measurements

“…For example, simulation techniques have been adapted to increase the volume and diversity of datasets for training deep learning models [5], where the position and texture of the object were randomized. On the other hand, sim-to-real techniques aim to transfer learning from simulations and adapt the model to the real environment [6]. However, less attention is paid to augmentation techniques for vision-based tactile sensors.…”