Learning Latent Space Dynamics for Tactile Servoing

Sutanto, Giovanni; Ratliff, Nathan; Sundaralingam, Balakumar; Chebotar, Yevgen; Su, Zhe; Handa, Ankur; Fox, Dieter

doi:10.1109/icra.2019.8793520

Cited by 28 publications

(19 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To our knowledge, no prior work has used video prediction models together with touch sensing for touch-based object repositioning. Concurrent work [20] learned a two dimensional latent space and dynamics model to perform control for following human demonstrations given in the tactile space, however handling of objects has not been shown yet.…”

Section: Related Workmentioning

confidence: 99%

Manipulation by Feel: Touch-Based Control with Deep Predictive Models

Tian

Ebert

Jayaraman

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

114

View full text Add to dashboard Cite

Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging. General purpose control techniques that are able to effectively leverage tactile sensing as well as accurate physics models of contacts and forces remain largely elusive, and it is unclear how to even specify a desired behavior in terms of tactile percepts. In this paper, we take a step towards addressing these issues by combining high-resolution tactile sensing with data-driven modeling using deep neural network dynamics models. We propose deep tactile MPC, a framework for learning to perform tactile servoing from raw tactile sensor inputs, without manual supervision. We show that this method enables a robot equipped with a GelSight-style tactile sensor to manipulate a ball, analog stick, and 20-sided die, learning from unsupervised autonomous interaction and then using the learned tactile predictive model to reposition each object to user-specified configurations, indicated by a goal tactile reading. Videos, visualizations and the code are available here: https://sites.google.com/view/deeptactilempcEqual contribution

show abstract

Section: Related Workmentioning

confidence: 99%

Manipulation by Feel: Touch-Based Control with Deep Predictive Models

Tian

Ebert

Jayaraman

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

114

View full text Add to dashboard Cite

show abstract

“…Many of them are concerned with estimating the stability of a grasp before lifting an object [6,14], even suggesting a regrasp [60]. Only a few approaches learn entire manipulation policies through reinforcement only given haptic feedback [29,30,[61][62][63]65]. While [30] relies on raw force-torque feedback, [29,61,62] learn a low-dimensional representation of high-dimensional tactile data before learning a policy, and [63] learns a dynamics model of the tactile feedback in a latent space.…”

Section: A Contact-rich Manipulationmentioning

confidence: 99%

Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

et al. 2020

View full text Add to dashboard Cite

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. It is non-trivial to manually design a robot controller that combines these modalities which have very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. In this work, we use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. Evaluating our method on a peg insertion task, we show that it generalizes over varying geometries, configurations, and clearances, while being robust to external perturbations. We also systematically study different self-supervised learning objectives and representation learning architectures. Results are presented in simulation and on a physical robot.

show abstract

“…This data can be measured with high accuracy and most of the probability distributions of the errors of these measurements are the same hence they mainly come from the physical construction of the manipulator. That is why, in demonstration learning tasks, usually other modalities are utilized apart from vision such as tactile information [101].…”

Section: ) Imitation and Demonstration-based Learningmentioning

confidence: 99%

Deep Learning in Robotics: Survey on Model Structures and Training Strategies

Karoly

Galambos

Kuti

et al. 2021

IEEE Trans. Syst. Man Cybern, Syst.

View full text Add to dashboard Cite

The ever-increasing complexity of robot applications induces the need for methods to approach problems with no (viable) analytical solution. Deep learning (DL) provides a set of tools to address this kind of problems. This survey presents a categorization of the major challenges in robotics that leverage DL technologies and introduces representative examples of successful solutions for the described problems. We also consider the question when and whether to use modular, monolithic models or end-to-end DL, in order to provide a guideline for the selection of the correct model structure and training strategy. By doing so, the current role and adaptability of different techniques at different hierarchical levels of a robot-application can be highlighted, thus providing a well-structured basis to assist future approaches. Index Terms-Deep learning (DL), machine learning (ML), manipulators, mobile robots, neural networks, robot control, robot learning. I. INTRODUCTION C OMPUTERS can easily solve formal problems that are demanding for humans. However, the increasing need for adaptive systems requires the solution of tasks that are hard to formulate, but can be easily solved by humans, such as the recognition and manipulation of objects. In order to perform such tasks, a certain complex knowledge of the environment is inevitable. The automatic extraction of the required knowledge is called machine learning (ML). The way the data is presented to the ML system, heavily influences how well the extracted knowledge represents the given problem. The ML approaches that also perform feature extraction, using multiple hierarchical artificial neural network layers, are referred to as deep learning (DL) [1], [2]. The theoretical background of DL had been introduced for a long time, when it finally gained widespread popularity, i.a., thanks to the winner entry of the ImageNet Challenge Manuscript

show abstract

Learning Latent Space Dynamics for Tactile Servoing

Cited by 28 publications

References 15 publications

Manipulation by Feel: Touch-Based Control with Deep Predictive Models

Manipulation by Feel: Touch-Based Control with Deep Predictive Models

Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

Deep Learning in Robotics: Survey on Model Structures and Training Strategies

Contact Info

Product

Resources

About