Hardware Technology of Vision-Based Tactile Sensor: A Review

Zhang, Shixin; Chen, Zixi; Gao, Yuan; Wan, Weiwei; Shan, Jiayuan; Xue, Hongtao; Sun, Fuchun; Yang, Yiyong; Fang, Bin

doi:10.1109/jsen.2022.3210210

Cited by 76 publications

(27 citation statements)

References 122 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Nonetheless, designing tactile sensors faced complexity in system integration and data processing, since increasing the scale requires a great deal of embedded sensing elements. Recently, vision-based tactile (ViTac) sensors have emerged as an effective method for the implementation of tactile sensing with a simple design [6], [7], [8]. In detail, the deformation of soft artificial skins upon physical contact with an object is detected through the optical tracking of visual features, such as markers or reflective membranes, which is then translated into tactile information, including contact location, force, vibration, object texture, and so on.…”

mentioning

confidence: 99%

Simulation, Learning, and Application of Vision-Based Tactile Sensing at Large Scale

Luu

Nguyen

2023

IEEE Trans. Robot.

View full text Add to dashboard Cite

Large-scale robotic skin with tactile sensing ability is emerging with the potential for use in close-contact human-robot systems. Although recent developments in vision-based tactile sensing and related learning methods are promising, they have been mostly designed for small-scale use, such as by fingers and hands, in manipulation tasks. Moreover, learning perception for such tactile devices demands a huge tactile dataset, which complicates the data collection process. To address this, this study introduces a multiphysics simulation pipeline, called SimTacLS, which considers not only the mechanical properties of external physical contact but also the realistic rendering of tactile images in a simulation environment. The system utilizes the obtained simulation dataset, including virtual images and skin deformation, to train a tactile deep neural network to extract high-level tactile information. Moreover, we adopt a generative network to minimize sim2real inaccuracy, preserving the simulation-based tactile sensing performance. Last but not least, we showcase this sim2real sensing method for our large-scale tactile sensor (TacLink) by demonstrating its use in two trial cases, namely, whole-arm nonprehensile manipulation and intuitive motion guidance, using a custom-built tactile robot arm integrated with TacLink. This article opens new possibilities in the learning of transferable tactile-driven robotics tasks from virtual worlds to actual scenarios without compromising accuracy.

show abstract

mentioning

confidence: 99%

Simulation, Learning, and Application of Vision-Based Tactile Sensing at Large Scale

Luu

Nguyen

2023

IEEE Trans. Robot.

View full text Add to dashboard Cite

show abstract

“…In 2D MDM, there are two frequently-used approaches to extract the deformation: obtain each marker's displacement (or coordinate) through recognition and tracking, or directly use the 2D tactile image as the input for end-to-end learning. a) Marker recognition and tracking In a real sensor, the markers are a series of objects with a certain size [16]. The GelSight sensor used ink dots as markers [24], the GelForce sensor embedded markers of two colors (red and blue) [27], and the TacTip sensor used pin-shaped markers distributed on the inner wall [34].…”

Section: B Technologies and Implementationmentioning

confidence: 99%

“…By constructing the mechanical model of the soft elastomer and designing the corresponding post-processing algorithm, it is viable to achieve the multiple tactile sensory, e.g., surface shape recognition [18], force-field measurement [19], contact region estimation [20], and slippage detection [21]. Noteworthily, in current studies, the reconstruction and recognition are commonly achieved by the mapping relationship from the deformation information to each contact modal [12], [16]. Therefore, the deformation of the contact surface is typically regarded as the original tactile information that can be used to reconstruct other contact features.…”

Section: Introductionmentioning

confidence: 99%

Marker Displacement Method Used in Vision-Based Tactile Sensors—From 2-D to 3-D: A Review

Jiang

2023

IEEE Sensors J.

View full text Add to dashboard Cite

The vision-based tactile sensor has been proven to be a promising device for sensing tactile information. Among such sensors, the marker displacement method (MDM) is the most common method used in such sensors for representing and extracting contact information. It uses the position field and displacement field of a marker array to characterize the original tactile information, and further achieves multimodal tactile perception through original information processing. This article is the first to classify MDM into three typical categories based on the dimensionality perspective: 2D MDM, 2.5D MDM, and 3D MDM. A comparison study is presented with a focus on the principles, characteristics, applications, and distinctions of these three methods. The latest literature has also been researched as the arguments. Finally, a summary of these three categories is presented as a helpful reference.

show abstract

“…It is challenging to simulate elastomers like gel layers in optical tactile sensors due to their deformation properties and high degrees of freedom. FEM has been widely applied to simulate deformation in [3], [10]. Nodes and facets are applied in [20] to represent a rubber sphere and a foam cube.…”

Section: Related Work a Elastomer Simulationmentioning

confidence: 99%

“…Tactile sensing is indispensable for robot contact control, and many tactile sensors have been developed in recent years [9], [10]. Thanks to the ability to produce high-resolution tactile images, optical tactile sensors [8], [11]- [13] that use cameras to capture deformation of soft elastomer have been favoured for robotics research.…”

mentioning

confidence: 99%

Tacchi: A Pluggable and Low Computational Cost Elastomer Deformation Simulator for Optical Tactile Sensors

Chen

Zhang

Luo

et al. 2023

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

Simulation is widely applied in robotics research to save time and resources. There have been several works to simulate optical tactile sensors that leverage either a smoothing method or Finite Element Method (FEM). However, elastomer deformation physics is not considered in the former method, whereas the latter requires a massive amount of computational resources like a computer cluster. In this work, we propose a pluggable and low computational cost simulator using the Taichi programming language for simulating optical tactile sensors, named as Tacchi. It reconstructs elastomer deformation using particles, which allows deformed elastomer surfaces to be rendered into tactile images and reveals contact information without suffering from high computational costs. Tacchi facilitates creating realistic tactile images in simulation, e.g., ones that capture wear-and-tear defects on object surfaces. In addition, the proposed Tacchi can be integrated with robotics simulators for a robot system simulation. Experiment results showed that Tacchi can produce images with better similarity to real images and achieved higher Sim2Real accuracy compared to the existing methods. Moreover, it can be connected with MuJoCo and Gazebo with only the requirement of 1G memory space in GPU compared to a computer cluster applied for FEM. With Tacchi, physical robot simulation with optical tactile sensors becomes possible. All the materials in this paper are available at https://github.com/zixichen007115/Tacchi.

show abstract

Hardware Technology of Vision-Based Tactile Sensor: A Review

Cited by 76 publications

References 122 publications

Simulation, Learning, and Application of Vision-Based Tactile Sensing at Large Scale

Simulation, Learning, and Application of Vision-Based Tactile Sensing at Large Scale

Marker Displacement Method Used in Vision-Based Tactile Sensors—From 2-D to 3-D: A Review

Tacchi: A Pluggable and Low Computational Cost Elastomer Deformation Simulator for Optical Tactile Sensors

Contact Info

Product

Resources

About