Hierarchical Neural Dynamic Policies

Abstract

We tackle the problem of generalization to unseen configurations for dynamic tasks in the real world while learning from high-dimensional image input. The family of nonlinear dynamical system-based methods have successfully demonstrated dynamic robot behaviors but have difficulty in generalizing to unseen configurations as well as learning from image inputs. Recent works approach this issue by using deep network policies and reparameterize actions to embed the structure of dynamical systems but still struggle in domains with diverse configurations of image goals, and hence, find it difficult to generalize. In this paper, we address this dichotomy by leveraging embedding the structure of dynamical systems in a hierarchical deep policy learning framework, called Hierarchical Neural Dynamical Policies (H-NDPs). Instead of fitting deep dynamical systems to diverse data directly, H-NDPs form a curriculum by learning local dynamical system-based policies on small regions in state-space and then distill them into a global dynamical system-based policy that operates only from high-dimensional images. H-NDPs additionally provide smooth trajectories, a strong safety benefit in the real world. We perform extensive experiments on dynamic tasks both in the real world (digit writing, scooping, and pouring) and simulation (catching, throwing, picking). We show that H-NDPs are easily integrated with both imitation as well as reinforcement learning setups and achieve state-of-the-art results. Video results are at https://shikharbahl.github.io/hierarchical-ndps/.

Train Offline, Test Online: A Real Robot Learning Benchmark

“…The task suite consists of two manipulation tasks that humans encounter every day, similar to those introduced in prior work [4], [25]. The tasks are pouring and scooping, excluding the easiest and hardest RB2 tasks (zipping and insertion).…”

A Survey on Physics Informed Reinforcement Learning: Review and Open Problems

“…Physics values extracted/ derived from the environ-ment or system has been directly used by RL agents in form of physics parameters [113], dynamic movement (physics) primitives [3], physical state [59] and physical target [75]. For example in [75], the reward is created to meet two physical objectives/ targets: operation cost and self-energy sustainability.…”

“…Extending this work to hierarchical deep policy learning framework, [3] introduced H-NDP which forms a curriculum by learning local dynamical system-based policies on small state-space region and then refines them into global dynamical system based policy. Given the accurate dynamics and constraint of the system [140] introduces control barrier certificates into actor-critic RL framework, for learning safe policies in dynamical systems.…”

Analyzing Neural Jacobian Methods in Applications of Visual Servoing and Kinematic Control

“…These models are time-discrete, which leads to performance degradation with high frequencies [3]. Other research proposes continuous-time approaches that use a deep model to generate the parameters of movement primitive models [3], [2], but the fundamental movements are still implemented with a linear model.…”