Zero-Shot Sim-to-Real Transfer of Tactile Control Policies for Aggressive Swing-Up Manipulation

Abstract

This letter aims to show that robots equipped with a vision-based tactile sensor can perform dynamic manipulation tasks without prior knowledge of all the physical attributes of the objects to be manipulated. For this purpose, a robotic system is presented that is able to swing up poles of different masses, radii and lengths, to an angle of 180 • , while relying solely on the feedback provided by the tactile sensor. This is achieved by developing a novel simulator that accurately models the interaction of a pole with the soft sensor. A feedback policy that is conditioned on a sensory observation history, and which has no prior knowledge of the physical features of the pole, is then learned in the aforementioned simulation. When evaluated on the physical system, the policy is able to swing up a wide range of poles that differ significantly in their physical attributes without further adaptation. To the authors' knowledge, this is the first work where a feedback policy from high-dimensional tactile observations is used to control the swing-up manipulation of poles in closed-loop.

Sim-to-Real for High-Resolution Optical Tactile Sensing: From Images to Three-Dimensional Contact Force Distributions

“…While this was not in the scope of this work, different simulation techniques can trade off accuracy to achieve real-time capabilities and become suitable to warm-start the training of tactile policies in simulation, as detailed in a related work. 35 …”

General-Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors

“…Compared with rigid-body-based simulation, Finite Element Methods (FEM) can model the deformation of the sensor's elastomer more accurately. Bi et al [29] developed a FEM-based tactile sensor simulator and achieved zero-shot Sim2Real transfer of RL policies for aggressive swing-up manipulation. However, they utilized the cylindrical geometry of the poles to simplify the simulation, and constrained the motion of the pole in the x-y plane, rendering it inapplicable to objects with various geometries.…”

Rotating without Seeing: Towards In-hand Dexterity through Touch

“…Another important question in tactile robotic manipulation is how to simulate the tactile event so as to perform Sim2Real transfer. Researchers have proposed many approaches and strategies for tactile simulation [40,24,73,18,61,9]. For example, Xu et al [73] propose a method to simulate normal and shear tactile force field on the contact surface.…”