Deep predictive policy training using reinforcement learning

Abstract

Skilled robot task learning is best implemented by predictive action policies due to the inherent latency of sensorimotor processes. However, training such predictive policies is challenging as it involves finding a trajectory of motor activations for the full duration of the action. We propose a data-efficient deep predictive policy training (DPPT) framework with a deep neural network policy architecture which maps an image observation to a sequence of motor activations. The architecture consists of three sub-networks referred to as the perception, policy and behavior super-layers. The perception and behavior super-layers force an abstraction of visual and motor data trained with synthetic and simulated training samples, respectively. The policy super-layer is a small subnetwork with fewer parameters that maps data in-between the abstracted manifolds. It is trained for each task using methods for policy search reinforcement learning. We demonstrate the suitability of the proposed architecture and learning framework by training predictive policies for skilled object grasping and ball throwing on a PR2 robot. The effectiveness of the method is illustrated by the fact that these tasks are trained using only about 180 real robot attempts with qualitative terminal rewards.

Asymmetric Actor Critic for Image-Based Robot Learning

“…Another approach is domain adaptation from the simulator [8,12,18,39,7,52], since it may be easier to fine tune from a simulator policy than training in the real world. However if the simulator differs from the real world, the policy trained in simulation can perform very poorly in the real world and fine tuning may not be any easier than training from scratch.…”

“…These physical robots could also damage themselves and their environment while exploring these behaviours. A recent approach to circumvent these challenges is to train on a simulated version of the robot and then transfer the learned policy to the real robot [8,12,18,39,7,52,40,13].…”

Self-Supervised Correspondence in Visuomotor Policy Learning

“…2) Ground truth 2D image coordinates (GT-2D): z is similar to the previous baseline, but the points are projected into the camera using the ground truth camera parameters. 3) Autoencoder (AE): z is the encoding of a pre-trained autoencoder, similar to the visual training in [2], [11]. 4) End-to-End (E2E): z is the intermediate representation from end-to-end training.…”

“…Note that the two vision-based baselines AE and E2E share an identical model architecture for producing z, and differ only in the method used to train the parameters. The model is close to [1]- [3] with the key architectural traits of having a few convolutional layers followed by a channel-wise spatial expectation operation, which has been widely used [10], [11], [25], [29], [36]- [38]. Most methods we compare (AE, E2E, DD-2D) use only one RGB camera stream as input to learned policies; DD-3D additionally uses the depth image.…”

A neural model for eye–head–arm coordination

“…Each PC/BC-DIM stage functions like a spatial auto-encoder which acted as an encoder for the transformation in one direction whereas the same PC/BC-DIM stage acted as a decoder for the transformation in the opposite direction. The PC/BC-DIM auto-encoder thus has a profound difference compared to all previously reported work for similar eyes-hand coordination tasks where the auto-encoder employed separate encoder and decoder neural circuitry [36][37][38][39].…”