Sim-to-real for high-resolution optical tactile sensing: From images to 3D contact force distributions

Abstract

The images captured by vision-based tactile sensors carry information about high-resolution tactile fields, such as the distribution of the contact forces applied to their soft sensing surface. However, extracting the information encoded in the images is challenging and often addressed with learningbased approaches, which generally require a large amount of training data. This article proposes a strategy to generate tactile images in simulation for a vision-based tactile sensor based on an internal camera that tracks the motion of spherical particles within a soft material. The deformation of the material is simulated in a finite element environment under a diverse set of contact conditions, and spherical particles are projected to a simulated image. Features extracted from the images are mapped to the 3D contact force distribution, with the ground truth also obtained via finite-element simulations, with an artificial neural network that is therefore entirely trained on synthetic data avoiding the need for real-world data collection. The resulting model exhibits high accuracy when evaluated on real-world tactile images, is transferable across multiple tactile sensors without further training, and is suitable for efficient real-time inference.

Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation

“…[15]- [22]), the authors are not aware of any work where a simulation from first principles is utilized to learn tactile feedback control policies. Rather, the mentioned works focus on gathering supervised datasets of tactile images in simulation to train deep neural networks that can predict object position and rotation ( [15], [20]), the force distribution acting on the sensor surface ( [21], [22]), or the three-dimensional mesh of the object in contact ( [16]).…”

“…In [21], a method to generate such images in simulation is presented to train a supervised learning architecture that aims to accurately estimate the real-world 3D contact force distribution. The same approach to generate training data is employed here, using finite-element simulations of the sensor surface under various contact conditions, where hyperelastic material models for the sensor's soft materials are employed.…”

“…The same approach to generate training data is employed here, using finite-element simulations of the sensor surface under various contact conditions, where hyperelastic material models for the sensor's soft materials are employed. The details of this procedure can be found in [21]. In addition to the two mentioned images per datapoint, the polar coordinates of each pixel are encoded here as two additional image channels.…”

“…On the real-world sensor, camera images are preprocessed to match those of the simulated training dataset as described in [21] and [22]. Specifically, images are converted to grayscale and remapped using the real-world camera model (obtained via a state-of-the-art calibration technique [26]) to images of the same scene as if they were taken in the simulated world.…”

Elastomer-Based Visuotactile Sensor for Normality of Robotic Manufacturing Systems

“…Gomes et al [ 32 ] introduced a synthetic VBTS system to support a Sim2Real learning process, which employed filters to construct the height map with different intensities. Sferrazzai et al [ 33 ] developed a simulation of VBTS that generates synthetic images of spherical particles in the sensor’s elastomer to reconstruct the 3D distribution of the contact forces. However, the sensor requires a large set of data to accurately sense the surface.…”