Towards Hierarchical Task Decomposition using Deep Reinforcement Learning for Pick and Place Subtasks

Abstract

Deep Reinforcement Learning (DRL) is emerging as a promising approach to generate adaptive behaviors for robotic platforms. However, a major drawback of using DRL is the data-hungry training regime that requires millions of trial and error attempts, which is impractical when running experiments on robotic systems. To address this issue, we propose a multi-subtask reinforcement learning method where complex tasks are decomposed manually into low-level subtasks by leveraging human domain knowledge. These subtasks can be parametrized as expert networks and learned via existing DRL methods. Trained subtasks can then be composed with a high-level choreographer. As a testbed, we use a pick and place robotic simulator to demonstrate our methodology, and show that our method outperforms an imitation learning-based method and reaches a high success rate compared to an endto-end learning approach. Moreover, we transfer the learned behavior in a different robotic environment that allows us to exploit sim-to-real transfer and demonstrate the trajectories in a real robotic system. Our training regime is carried out using a central processing unit (CPU)-based system, which demonstrates the data-efficient properties of our approach.

Reinforcement Learning Your Way: Agent Characterization through Policy Regularization

“…Hierarchical reinforcement learning HRL [13,14] divides agents' tasks into sub-tasks to be learned by different agents. This simplifies the problem to be solved by each agent, making their behaviour easier to interpret and thereby making them easier to characterize.…”

Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates

“…As a remedy the broad area of Hierarchical Reinforcement Learning (HRL) attempts to decompose RL problems into multiple levels of abstraction-temporal, spatial, or otherwise. Many works deploy separate policies over different time horizons and action spaces (Barto & Mahadevan, 2003;Levy et al, 2017;Yang et al, 2020;Marzari et al, 2021). Temporal abstraction in planning can be traced back at least to Sutton et al (1999), where the options were introduced to refer to lower level policies.…”

Reinforcement Learning with Intrinsic Affinity for Personalized Prosperity Management

“…Larger problems are solved by choreographing these subtasks through an orchestration agent that learns the highlevel dynamics of its environment [25]. HRL has, for instance, been used to control a robotic arm: while low-level agents learned simple tasks such as moving forward / backward or picking up / placing down, an orchestration agent learned to retrieve objects on a surface by choreographing these tasks [26,27]. The agents were not only efficient at learning, but their policies were more easily interpreted by human experts.…”