Hierarchical Reinforcement Learning With Universal Policies for Multistep Robotic Manipulation

Abstract

Multi-step tasks, such as block stacking or parts (dis)assembly, are complex for autonomous robotic manipulation. A robotic system for such tasks would need to hierarchically combine motion control at a lower level and symbolic planning at a higher level. Recently, reinforcement learning (RL) based methods have been shown to handle robotic motion control with better flexibility and generalisability. However, these methods have limited capability to handle such complex tasks involving planning and control with many intermediate steps over a long time horizon.Firstly, current RL systems cannot achieve varied outcomes by planning over intermediate steps (e.g., stacking blocks in different orders). Secondly, exploration efficiency of learning multi-step tasks is low, especially when rewards are sparse.To address these limitations, we develop a unified hierarchical reinforcement learning framework, named Universal Option Framework (UOF), to enable the agent to learn varied outcomes in multi-step tasks. To improve learning efficiency, we train both symbolic planning and kinematic control policies in parallel, aided by two proposed techniques: 1) an auto-adjusting exploration strategy (AAES) at the low level to stabilise the parallel training, and 2) abstract demonstrations at the high level to accelerate convergence.To evaluate its performance, we performed experiments on various multi-step block-stacking tasks with blocks of different shapes and combinations and with different degrees of freedom for robot control. The results demonstrate that our method can accomplish multi-step manipulation tasks more efficiently and stably, and with significantly less memory consumption.

An Open-Source Multi-goal Reinforcement Learning Environment for Robotic Manipulation with Pybullet

“…It is then potentially fruitful to develop a better curriculum for such tasks. Another interesting direction is to leverage task decomposition for multi-step tasks and make use of hierarchical learning systems [18]. The use of sub-goals is a promising way to tackle the hard exploration problem in such tasks.…”

“…To improve readability, the original set of tasks is named 'single-step tasks' and the new set of tasks is named 'multi-step tasks'. The multi-step tasks are developed with the aim to inspire new learning algorithms that can handle tasks where the reward signals only appear near the end of the task horizon [5,18]. Beside the delayed rewards, these tasks also require multiple steps to complete, and some of the steps are strongly dependent.…”

Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods

“…O VER the past few years, RL has achieved impressive success in a wide variety of tasks, including games [1]- [5] and robotic control [6]- [11]. Specifically, on continuous control, actor-critic RL algorithms [12]- [15] have been widely explored with remarkable performance.…”

Abstract Demonstrations and Adaptive Exploration for Efficient and Stable Multi-step Sparse Reward Reinforcement Learning

“…Inspired by this, we propose firstly to leverage abstract demonstrations, which provide the correct order of steps to a learning agent instead of low-level motions. It has been shown that, given an efficient enough algorithm to learn the subtasks such as hindsight experience replay, abstract demonstrations can accelerate learning for multi-step tasks [5]. The main benefits of abstract demonstrations when compared to demonstrations of motion trajectories [6], [7] are: 1) they do not encode a specific pattern of the behaviours when solving a task; 2) they release robotic operators from the tedious processes of collecting motion trajectory data.…”