Skill generalization of tubular object manipulation with tactile sensing and Sim2Real learning

Zhao, Yue; Jing, Xingshuo; Qian, Kun; Gomes, Daniel Fernandes; Luo, Shan

doi:10.1016/j.robot.2022.104321

Cited by 17 publications

(10 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, the access to ground-truth environment states in simulation accelerates the policy learning process. Most existing tactile sensor simulation methods are based on rigid-bod physics simulation [10], [11], [12], [13], [14]. While the penalty-based contact model can approximate the deformation of the tactile sensor, it cannot capture the elastic dynamics of the tactile sensor accurately.…”

Section: Zero-shot Sim2realmentioning

confidence: 99%

General-Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors

Chen,

Xu,

Xiang

et al. 2024

IEEE Trans. Robot.

View full text Add to dashboard Cite

Visuotactile sensors can provide rich contact information, having great potential in contact-rich manipulation tasks with reinforcement learning (RL) policies. Sim2Real technique tackles the challenge of RL's reliance on a large amount of interaction data. However, most Sim2Real methods for manipulation tasks with visuotactile sensors rely on rigidbody physics simulation, which fails to simulate the real elastic deformation precisely. Moreover, these methods do not exploit the characteristic of tactile signals for designing the network architecture. In this paper, we build a general-purpose Sim2Real protocol for manipulation policy learning with marker-based visuotactile sensors. To improve the simulation fidelity, we employ an FEM-based physics simulator that can simulate the sensor deformation accurately and stably for arbitrary geometries. We further propose a novel tactile feature extraction network that directly processes the set of pixel coordinates of tactile sensor markers and a self-supervised pre-training strategy to improve the efficiency and generalizability of RL policies. We conduct extensive Sim2Real experiments on the peg-in-hole task to validate the effectiveness of our method. And we further show its generalizability on additional tasks including plug adjustment and lock opening. The protocol, including the simulator and the policy learning framework, will be open-sourced for community usage.

show abstract

Section: Zero-shot Sim2realmentioning

confidence: 99%

General-Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors

Chen,

Xu,

Xiang

et al. 2024

IEEE Trans. Robot.

View full text Add to dashboard Cite

show abstract

“…Thanks to its simple fabrication and the high-resolution of tactile images for robots to extract rich information, a wide variety of designs have been proposed to use this working principle to construct robotic tactile sensors Dong et al [7], Lambeta et al [20], Donlon et al [8], Taylor et al [30]. Most of these designs focused on miniaturising the sensor and they have been extensively investigated for various applications Cao et al [3], Jiang et al [15], Lee et al [21], Jiang et al [16], Jing et al [18], Zhao et al [37], Calandra et al [2], Cao et al [4]. However, most of them are constrained with a single flat sensing area, which limits the potential of these sensors being used in unstructured environments where unexpected contacts can occur either inside or outside of the grasp closure Gomes et al [10].…”

Section: Related Work a High-resolution Optical Tactile Sensorsmentioning

confidence: 99%

“…Compared to running the entire work using the real setup, the Sim2Real approach is less damage-prone to the tactile sensors, and potentially less time-consuming by the usage of parallel training in simulation. To this end, in recent years simulation models have been proposed for camera-based tactile sensors, particularly GelSight sensors Gomes et al [9,12], Agarwal et al [1], Wang et al [33], Chen et al [5], which have been widely used Hogan et al [14], Zhao et al [37], Jiang et al [16] and are of our interest.…”

Section: Introductionmentioning

confidence: 99%

Beyond Flat GelSight Sensors: Simulation of Optical Tactile Sensors of Complex Morphologies for Sim2Real Learning

Gomes¹,

Luo²,

Paoletti³

2023

Robotics: Science and Systems XIX

View full text Add to dashboard Cite

Recently, several morphologies, each with its advantages, have been proposed for the GelSight high-resolution tactile sensors. However, existing simulation methods are limited to flatsurface sensors, which prevents its usage with the newer sensors of non-flat morphologies in Sim2Real experiments. In this paper, we extend a previously proposed GelSight simulation method, which was developed for flat-surface sensors, and propose a novel method for curved sensors. In particular, we address the simulation of light rays travelling through a curved tactile membrane in the form of geodesic paths. The method is validated by simulating the finger-shaped GelTip sensor and comparing the generated synthetic tactile images against the corresponding real images. Our extensive experiments show that combining the illumination generated from the geodesic paths, with a background image from the real sensor, produces the best results when compared to the lighting generated by direct linear paths in the same conditions. As the method is parameterised by the sensor mesh, it can be applied in principle to simulate a tactile sensor of any morphology. The proposed method not only unlocks simulating existing optical tactile sensors of complex morphologies, but also enables experimenting with sensors of novel morphologies, before the fabrication of the real sensor.

show abstract

“…In a later work, an improved CycleGAN architecture was introduced with taskspecific loss functions for enhanced structural fidelity of generated tactile images [25]. However, these approaches and others [26], [27] focused on a specific tactile sensor with a flat contact surface and without exhibiting zero-shot capability. In addition, prior work focused on tactile images where the contact trace seen in the image is rather large [1].…”

Section: Introductionmentioning

confidence: 99%

Learning Haptic-Based Object Pose Estimation for In-Hand Manipulation Control With Underactuated Robotic Hands

Azulay

Ben-David

Sintov

2023

IEEE Trans. Haptics

View full text Add to dashboard Cite

Simulating tactile perception could potentially leverage the learning capabilities of robotic systems in manipulation tasks. However, the reality gap of simulators for high-resolution tactile sensors remains large. Models trained on simulated data often fail in zero-shot inference and require fine-tuning with real data. In addition, work on high-resolution sensors commonly focus on ones with flat surfaces while 3D round sensors are essential for dexterous manipulation. In this paper, we propose a bi-directional Generative Adversarial Network (GAN) termed SightGAN. SightGAN relies on the early CycleGAN while including two additional loss components aimed to accurately reconstruct background and contact patterns including small contact traces. The proposed SightGAN learns real-to-sim and sim-to-real processes over difference images. It is shown to generate real-like synthetic images while maintaining accurate contact positioning. The generated images can be used to train zero-shot models for newly fabricated sensors. Consequently, the resulted sim-toreal generator could be built on top of the tactile simulator to provide a real-world framework. Potentially, the framework can be used to train, for instance, reinforcement learning policies of manipulation tasks. The proposed model is verified in extensive experiments with test data collected from real sensors and also shown to maintain embedded force information within the tactile images.

show abstract

Skill generalization of tubular object manipulation with tactile sensing and Sim2Real learning

Cited by 17 publications

References 21 publications

General-Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors

General-Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors

Beyond Flat GelSight Sensors: Simulation of Optical Tactile Sensors of Complex Morphologies for Sim2Real Learning

Learning Haptic-Based Object Pose Estimation for In-Hand Manipulation Control With Underactuated Robotic Hands

Contact Info

Product

Resources

About