Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing

Sferrazza, Carmelo; Bi, Thomas; D’Andrea, Raffaello

doi:10.1109/iros45743.2020.9341285

Cited by 29 publications

(27 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, the reconstruction of surface normals requires evenly distributed light, without shadows or internal reflections (21). Tracking markers (23,33,39) rather than a surface does not solve the fundamental problems with displacement-focused approaches.…”

Section: Methodsmentioning

confidence: 99%

“…Further developments of this approach increased its robustness (GelSlim ( 22)), achieved curved sensing surfaces with one camera (GelTip (28)) and with five cameras (OmniTact ( 27)), and included markers to obtain shear force information (22). A different technique based on tracking of small beads inside a transparent elastomer is used by GelForce (33) and the Sferrazza and D'Andrea sensor (23,39) to estimate normal and shear force maps. ChromaForce (not listed in the table) uses subtractive color mixing to extract similar data from deformable optical markers in a transparent layer (55).…”

Section: Operating Speedmentioning

confidence: 99%

See 1 more Smart Citation

A soft thumb-sized vision-based sensor with accurate all-round force perception

Sun¹,

Kuchenbecker²,

Martius³

2021

Preprint

View full text Add to dashboard Cite

Vision-based haptic sensors have emerged as a promising approach to robotic touch due to affordable high-resolution cameras and successful computer-vision techniques. However, their physical design and the information they provide do not yet meet the requirements of real applications. We present a robust, soft, low-cost, vision-based, thumb-sized 3D haptic sensor named Insight: it continually provides a directional force-distribution map over its entire conical sensing surface. Constructed around an internal monocular camera, the sensor has only a single layer of elastomer over-molded on a stiff frame to guarantee sensitivity, robustness, and soft contact. Furthermore, Insight is the first system to combine photometric stereo and structured light using a collimator to detect the 3D deformation of its easily replaceable flexible outer shell. The force information is inferred by a deep neural network that maps images to the spatial distribution of 3D contact force (normal and shear). Insight has an overall spatial resolution of 0.4 mm, force magnitude accuracy around 0.03 N, and force direction accuracy around 5 degrees over a range of 0.03-2 N for numerous distinct contacts with varying contact area. The presented hardware and software design concepts can be transferred to a wide variety of robot parts.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Operating Speedmentioning

confidence: 99%

A soft thumb-sized vision-based sensor with accurate all-round force perception

Sun¹,

Kuchenbecker²,

Martius³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Motivated from the ability of current learning based techniques to process higher dimensional data such as images, recent approaches such as Baghaei Naeini et al ( 2020 ), She et al ( 2020 ), Sferrazza et al ( 2020 ), Han et al ( 2018 ), and Liang et al ( 2018 ) aim to employ deep learning based architectures to predict contact forces and stress maps. A few of the approaches rely on sensors (Han et al, 2018 ; She et al, 2020 ; Thuruthel et al, 2019 ) to predict the forces from sensory feedback, however, there is a limit to the amount of data which can be recorded, along with effecting the compliance of the gripper and the difficulty in wiring.…”

Section: Current Literaturementioning

confidence: 99%

“…However, such sensors could not be used with FRE grippers due to the flexibility of the grippers. Alternative approach is to employ a vision based system such as a tactile sensor (Sferrazza et al, 2020 ) or a dynamic vision system (Baghaei Naeini et al, 2020 ) on gripper which observes the indentation of a deformable surface, rather than the gripper/compliant structure. The system utilises a machine learning system, trained via simulated data in order to estimate a contact force map.…”

Section: Current Literaturementioning

confidence: 99%

“…Liang et al ( 2018 ) use FEA to predict stress maps in human organs, however they only showcase the results on simulation data. Similarly, for force estimation (Sferrazza et al, 2020 ) use simulation data from FEA to train their deep network. They present prediction results for real-world data, however only limit to cylindrical objects.…”

Section: Current Literaturementioning

confidence: 99%

See 1 more Smart Citation

A Deep Learning Method for Vision Based Force Prediction of a Soft Fin Ray Gripper Using Simulation Data

Barrie

Pandya²,

Pandya

et al. 2021

Front. Robot. AI

View full text Add to dashboard Cite

Soft robotic grippers are increasingly desired in applications that involve grasping of complex and deformable objects. However, their flexible nature and non-linear dynamics makes the modelling and control difficult. Numerical techniques such as Finite Element Analysis (FEA) present an accurate way of modelling complex deformations. However, FEA approaches are computationally expensive and consequently challenging to employ for real-time control tasks. Existing analytical techniques simplify the modelling by approximating the deformed gripper geometry. Although this approach is less computationally demanding, it is limited in design scope and can lead to larger estimation errors. In this paper, we present a learning based framework that is able to predict contact forces as well as stress distribution from soft Fin Ray Effect (FRE) finger images in real-time. These images are used to learn internal representations for deformations using a deep neural encoder, which are further decoded to contact forces and stress maps using separate branches. The entire network is jointly learned in an end-to-end fashion. In order to address the challenge of having sufficient labelled data for training, we employ FEA to generate simulated images to supervise our framework. This leads to an accurate prediction, faster inference and availability of large and diverse data for better generalisability. Furthermore, our approach is able to predict a detailed stress distribution that can guide grasp planning, which would be particularly useful for delicate objects. Our proposed approach is validated by comparing the predicted contact forces to the computed ground-truth forces from FEA as well as real force sensor. We rigorously evaluate the performance of our approach under variations in contact point, object material, object shape, viewing angle, and level of occlusion.

show abstract

Minsight: A Fingertip‐Sized Vision‐Based Tactile Sensor for Robotic Manipulation

Andrussow

Sun

Kuchenbecker

et al. 2023

Advanced Intelligent Systems

View full text Add to dashboard Cite

Intelligent interaction with the physical world requires perceptual abilities beyond vision and hearing; vibrant tactile sensing is essential for autonomous robots to dexterously manipulate unfamiliar objects or safely contact humans. Therefore, robotic manipulators need high-resolution touch sensors that are compact, robust, inexpensive, and efficient. The soft vision-based haptic sensor presented herein is a miniaturized and optimized version of the previously published sensor Insight. Minsight has the size and shape of a human fingertip and uses machine learning methods to output high-resolution maps of 3D contact force vectors at 60 Hz. Experiments confirm its excellent sensing performance, with a mean absolute force error of 0.07 N and contact location error of 0.6 mm across its surface area. Minsight's utility is shown in two robotic tasks on a 3-DoF manipulator. First, closed-loop force control enables the robot to track the movements of a human finger based only on tactile data. Second, the informative value of the sensor output is shown by detecting whether a hard lump is embedded within a soft elastomer with an accuracy of 98%. These findings indicate that Minsight can give robots the detailed fingertip touch sensing needed for dexterous manipulation and physical human-robot interaction.

show abstract

Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing

Cited by 29 publications

References 15 publications

A soft thumb-sized vision-based sensor with accurate all-round force perception

A soft thumb-sized vision-based sensor with accurate all-round force perception

A Deep Learning Method for Vision Based Force Prediction of a Soft Fin Ray Gripper Using Simulation Data

Minsight: A Fingertip‐Sized Vision‐Based Tactile Sensor for Robotic Manipulation

Contact Info

Product

Resources

About