Reinforcement and Imitation Learning for Diverse Visuomotor Skills

Abstract

We propose a model-free deep reinforcement learning method that leverages a small amount of demonstration data to assist a reinforcement learning agent. We apply this approach to robotic manipulation tasks and train end-to-end visuomotor policies that map directly from RGB camera inputs to joint velocities. We demonstrate that our approach can solve a wide variety of visuomotor tasks, for which engineering a scripted controller would be laborious. In experiments, our reinforcement and imitation agent achieves significantly better performances than agents trained with reinforcement learning or imitation learning alone. We also illustrate that these policies, trained with large visual and dynamics variations, can achieve preliminary successes in zero-shot sim2real transfer. A brief visual description of this work can be viewed in this video.

Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future

“…In these systems, an artificial agent must learn to produce action outcomes in response to information from the environment, including rewards. Several such artificial systems have used convolutional architectures on the front end in order to process visual information about the world (Figure 3) [72,73,74]. It would be interesting to compare the representations learned in the context of these models to those trained by other mechanisms, as well as to data.…”

Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

“…Reinforcement learning approaches have recently been proposed to address variations in geometry and configuration for manipulation. [34,58] trained neural network policies using RGB images and proprioceptive feedback. Their approach works well in a wide range of tasks, but the large object clearances compared to automation tasks may explain the sufficiency of RGB data.…”

“…While the utility of multimodal data has frequently been shown in robotics [7,45,48,54], the proposed manipulation strategies are often task-specific. While learning-based methods do not require manual task specification, the majority of learned manipulation policies close the control loop around a single modality, often vision [15,21,34,58].…”

Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

“…Reinforcement learning approaches have recently been proposed to address variations in geometry and configuration for manipulation. [41,72] train neural network policies using RGB images and proprioceptive feedback. Their approach works well in a wide range of tasks, but the large object clearances compared to manufacturing tasks may explain the sufficiency of RGB data.…”

“…This makes many of these methods task-specific. On the other hand, most learning-based methods do not require manual task specification, yet the majority of learned manipulation policies close the control loop around a single modality, often RGB images [15,22,41,72].…”