Simulation, Learning, and Application of Vision-Based Tactile Sensing at Large Scale

Abstract

Large-scale robotic skin with tactile sensing ability is emerging with the potential for use in close-contact human-robot systems. Although recent developments in vision-based tactile sensing and related learning methods are promising, they have been mostly designed for small-scale use, such as by fingers and hands, in manipulation tasks. Moreover, learning perception for such tactile devices demands a huge tactile dataset, which complicates the data collection process. To address this, this study introduces a multiphysics simulation pipeline, called SimTacLS, which considers not only the mechanical properties of external physical contact but also the realistic rendering of tactile images in a simulation environment. The system utilizes the obtained simulation dataset, including virtual images and skin deformation, to train a tactile deep neural network to extract high-level tactile information. Moreover, we adopt a generative network to minimize sim2real inaccuracy, preserving the simulation-based tactile sensing performance. Last but not least, we showcase this sim2real sensing method for our large-scale tactile sensor (TacLink) by demonstrating its use in two trial cases, namely, whole-arm nonprehensile manipulation and intuitive motion guidance, using a custom-built tactile robot arm integrated with TacLink. This article opens new possibilities in the learning of transferable tactile-driven robotics tasks from virtual worlds to actual scenarios without compromising accuracy.

ELTac: A Vision-Based Electroluminescent Tactile Sensing Skin for Force Localization and Magnitude Estimation

“…The localization error in our work is 6.22-7.19mm. This range is comparable to that in Luu et al [40] and to the mesh size of 18mm × 9.5mm employed for localization in Duong et al [38] . To the best of our knowledge, very few studies in literature focused on force estimation in large area tactile sensing.…”

“…The methodology in this paper is limited to normal force estimation and is not suitable for shear force sensing. In contrast, the other methodologies based on marker tracking [35][36][37][38][39][40], in principle, allow for complete mapping of the deformation field and hence the potential for directional and shear forces. In summary, the ELTac sensing module offers high localization and force estimation accuracies as well as multi-point contact detection capabilities all enabled by a single camera.…”

“…On the basis of [38], [39] changed the skin to a polymer dispersed liquid crystal film, which can switch between the opaque and the transparent state, to collect tactile and proximity perception data in two modes. Luu et al [40] proposed a pipeline named SimTacLS to analyze the dataset captured by the visual tactile sensor in the TacLink device in order to carry out tactiledriven robotics tasks. In these studies, the internal lighting system plays a crucial role in detecting the markers accurately.…”

General-Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors

“…Narang et al built a linear-FEM-based tactile simulator for BioTac with Isaac Gym [31], achieving faster speeds than the commercial FEM software (ANSYS) [32]. Recently, Luu et al employed SOFA [33] to build a simulator for large-scale marker-cumvision-based-tactile sensor [34]. However, both [31] and [34] primarily used their simulators to collect supervised datasets for interpreting tactile signals, leaving the potential of using simulation to train manipulation policies unexplored.…”

Hardness recognition of robotic forearm based on visual–tactile fusion

“…The rapid development of vision technology has promoted the hardness recognition [6] of objects, but with the rapid development, its vision technology also possesses certain limitations, when recognizing objects with similar appearance and shape, such as plasticine and sand with the same color, it is more difficult to recognize objects through vision technology, Althoefer K et al [7] proposed to use a miniature tactile sensor, which is based on multiple gradients of force, and the change of force and shape of the object to obtain tactile information, and input the collected tactile information into a deep learning network to process the collected feature information. Luu Q K et al [8] proposed a new tactile algorithm network, which simulates the data set to train the tactile neural network to extract deep tactile information through the tactile network. Sundaram et al [9] demonstrated that an array of piezoresistive pressure sensors was assembled on a knitted glove to produce a tactile data set by contacting twenty-six types of objects, and the collected tactile data were fed into a deep convolutional neural network [10] to obtain the hardness classification of the objects [11].…”