Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration

Abstract

Deep Reinforcement Learning has been successfully applied to learn robotic control. However, the corresponding algorithms struggle when applied to problems where the agent is only rewarded after achieving a complex task. In this context, using demonstrations can significantly speed up the learning process, but demonstrations can be costly to acquire. In this paper, we propose to leverage a sequential bias to learn control policies for complex robotic tasks using a single demonstration. To do so, our method learns a goal-conditioned policy to control a system between successive low-dimensional goals. This sequential goal reaching approach raises a problem of compatibility between successive goals: we need to ensure that the state resulting from reaching a goal is compatible with the achievement of the following goals. To tackle this problem, we present a new algorithm called DCIL-II. We show that DCIL-II can solve with unprecedented sample efficiency some challenging simulated tasks such as humanoid locomotion and stand-up as well as fast running with a simulated Cassie robot. Our method leveraging sequentiality is a step towards the resolution of complex robotic tasks under minimal specification effort, a key feature for the next generation of autonomous robots.

Divide & Conquer Imitation Learning

“…To evaluate the drop in efficiency induced by resetting the agent to a single state, we compare SR-DCIL to DCIL-II. Indeed, by resetting the agent to demonstrated states, DCIL-II not only overcomes the limits underlined in Section IV-A, but it also learns a complex behavior by training on short rollouts only [2]. Therefore, DCIL-II should be more sample efficient than RF-DCIL at learning complex behaviors.…”

“…As an extension of DCIL-II [2], SR-DCIL extracts a sequence of goals from a demonstration and learns to reach them sequentially to reproduce the complex demonstrated behavior. As explained in Section II-A, this strategy is adopted by various classes of RL algorithms.…”

“…It is therefore important to reach each goal by passing through states from which it is possible to reach the following ones, or to select a specific goal sequence that the agent will be able to chain together. Value propagation mechanisms [1,2,9] can be used to achieve both these objectives. In DCIL, this mechanism requires a strong reset assumption.…”

“…The Divide & Conquer Imitation Learning algorithms [1,2] are novel Deep Reinforcement Learning (DRL) algorithms that leverage a sequential bias to learn a control policy for complex robotic tasks using a single demonstration. This approach can be used to learn a goal-conditioned policy to control the system between successive intermediate low-dimensional goals.…”

“…In this paper, we propose an extension of the DCIL-II algorithm [2] called Single-Reset DCIL (SR-DCIL). As DCIL-II, SR-DCIL learns a goal-conditioned policy to control the system between a sequence of low-dimensional goals.…”