Deep intrinsically motivated continuous actor-critic for efficient robotic visuomotor skill learning

Abstract

In this paper, we present a new intrinsically motivated actor-critic algorithm for learning continuous motor skills directly from raw visual input. Our neural architecture is composed of a critic and an actor network. Both networks receive the hidden representation of a deep convolutional autoencoder which is trained to reconstruct the visual input, while the centre-most hidden representation is also optimized to estimate the state value. Separately, an ensemble of predictive world models generates, based on its learning progress, an intrinsic reward signal which is combined with the extrinsic reward to guide the exploration of the actor-critic learner. Our approach is more data-efficient and inherently more stable than the existing actor-critic methods for continuous control from pixel data. We evaluate our algorithm for the task of learning robotic reaching and grasping skills on a realistic physics simulator and on a humanoid robot. The results show that the control policies learned with our approach can achieve better performance than the compared state-of-the-art and baseline algorithms in both dense-reward and challenging sparse-reward settings.

What Can Computational Models Learn From Human Selective Attention? A Review From an Audiovisual Unimodal and Crossmodal Perspective

“…When human toddlers detect any conflict between the current environment and their prior knowledge, they will generate curiosity and be motivated to learn new knowledge or rules (Wu and Miao, 2013). Curiosity is also important for the trial and error learning of robots (Hafez et al, 2019). In this subsection, we mainly talk about behavioral and neural mechanisms of selective attention underlying audiovisual crossmodal integration and conflict resolution.…”

BND*-DDQN: Learn to Steer Autonomously Through Deep Reinforcement Learning

“…In [61], the reward signals were augmented by intrinsic motivation to learn laser-based navigation policies for mobile robots. And it was also demonstrated in [62] that the intrinsic motivation can be combined with dense extrinsic reward to improve the learning of reaching and grasping skills. In this paper, we adopt the random network distillation bonus as the intrinsic reward and incorporate it into our training framework for performance improvement.…”

Teaching NICO How to Grasp: An Empirical Study on Crossmodal Social Interaction as a Key Factor for Robots Learning From Humans

“…Therefore, extensions to the DDPG algorithm and related algorithms have been suggested to enhance the sample efficiency and reduce the required training episodes. These approaches leverage the principles of intrinsic curiosity (Hafez et al, 2019), imagination (Andrychowicz et al, 2017), and task simplification (Kerzel et al, 2018). However, the basic problem of reinforcement learning of possibly unproductive and harmful exploratory actions remains.…”

“…Several approaches for grasp learning have been evaluated on NICO: Hafez et al (2017Hafez et al ( , 2019 successfully evaluated curiositydriven reinforcement learning both on a simulated and on a physical NICO. For the physical experiments, full training of the deep RL approach was conducted without human supervision for over 50 h during which NICO performed arm movements and grasp actions.…”