Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments

Goecks, Vinicius G.; Gremillion, Gregory M.; Lawhern, Vernon J.; Valasek, John; Waytowich, Nicholas R.

doi:10.48550/arxiv.1910.04281

Cited by 3 publications

(3 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, agents can learn from natural language-defined goals [185]. Finally, agents can learn from combining human data with reinforcement learning [137,59,139].…”

Section: Robotic Navigationmentioning

confidence: 99%

Learning Mobile Manipulation

Watkins¹

2022

Preprint

View full text Add to dashboard Cite

Providing mobile robots with the ability to manipulate objects has, despite decades of research, remained a challenging problem. The problem is approachable in constrained environments where there is ample prior knowledge of the environment layout and manipulatable objects. The challenge is in building systems that scale beyond specific situational instances and gracefully operate in novel conditions. In the past, researchers used heuristic and simple rule-based strategies to accomplish tasks such as scene segmentation or reasoning about occlusion. These heuristic strategies work in constrained environments where a roboticist can make simplifying assumptions about everything from the geometries of the objects to be interacted with, level of clutter, camera position, lighting, and a myriad of other relevant variables. The work in this thesis will demonstrate how to build a system for robotic mobile manipulation that is robust to changes in these variables. This robustness will be enabled by recent simultaneous advances in the fields of big data, deep learning, and simulation. The ability of simulators to create realistic sensory data enables the generation of massive corpora of labeled training data for various grasping and navigation-based tasks. It is now possible to build systems that work in the real world trained using deep learning entirely on synthetic data. The ability to train and test on synthetic data allows for quick iterative development of new perception, planning and grasp execution algorithms that work in many environments.To build a robust system, this thesis introduces a novel multiple-view shape reconstruction architecture that leverages unregistered views of the object. To navigate to objects without localizing the agent, this thesis introduces a novel panoramic target goal architecture that takes previous views of the agent to inform a policy to navigate through an environment. Additionally, a novel next-best-view methodology is introduced to allow the agent to move around the object and refine its initial understanding of the object. The results show that this deep learned sim-to-real approach performs best when compared to heuristic-based methods in terms of reconstruction quality and success-weighted-by-path-length (SPL). This approach is also adaptable to the environment and robot chosen due to its modular design.I would like to acknowledge all members of the Columbia Robotics Lab (CRLab) for making my stay there an enjoyable and rewarding experience. In particular, I would like to thank

show abstract

“…Additionally, agents can learn from natural language-defined goals [185]. Finally, agents can learn from combining human data with reinforcement learning [137,59,139].…”

Section: Robotic Navigationmentioning

confidence: 99%

Learning Mobile Manipulation

Watkins¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…First, SAC Behavioral Cloning (SACBC) uses Behavioral Cloning [20] as a regularization mechanism for the update of the actor. This is inspired by various other RLfD algorithms [21,22,23]. These methods all propose two main components.…”

Section: B Guiding Rl Using Demonstrationsmentioning

confidence: 99%

Divide & Conquer Imitation Learning

Chenu

Perrin-Gilbert

Sigaud

2022

2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Demonstrations are commonly used to speed up the learning process of deep reinforcement learning algorithms. To cope with the difficulty of accessing multiple demonstrations, some algorithms have been developed to learn from a single demonstration. In particular, the Divide & Conquer Imitation Learning algorithms leverage a sequential bias to learn a control policy for complex robotic tasks using a single state-based demonstration. The latest version, DCIL-II demonstrates remarkable sample efficiency. This novel method operates within an extended Goal-Conditioned Reinforcement Learning framework, ensuring compatibility between intermediate and subsequent goals extracted from the demonstration. However, a fundamental limitation arises from the assumption that the system can be reset to specific states along the demonstrated trajectory, confining the application to simulated systems. In response, we introduce an extension called Single-Reset DCIL (SR-DCIL), designed to overcome this constraint by relying on a single initial state reset rather than sequential resets. To address this more challenging setting, we integrate two mechanisms inspired by the Learning from Demonstrations literature, including a Demo-Buffer and Value Cloning to guide the agent toward compatible success states. In addition, we introduce Approximate Goal Switching to facilitate training to reach goals distant from the reset state. Our paper makes several contributions, highlighting the importance of the reset assumption in DCIL-II, presenting the mechanisms of SR-DCIL variants and evaluating their performance in challenging robotic tasks compared to DCIL-II. In summary, this work offers insights into the significance of reset assumptions in the framework of DCIL and proposes SR-DCIL, a first step toward a versatile algorithm capable of learning control policies under a weaker reset assumption.

show abstract

“…They introduced a Behavior Cloning (BC) loss to the DRL optimization function and showed that the agent is more efficient than a baseline DDPG method. Similarly, Goecks et al [12] proposed a two-phase combination of BC and DRL, where demonstrations were used to pretrain the network followed by training a DRL agent to produce an adaptable behavior. In this paper, we experiment with the latter for training expert subtasks.…”

Section: Related Workmentioning

confidence: 99%

Towards Hierarchical Task Decomposition using Deep Reinforcement Learning for Pick and Place Subtasks

Marzari

Pore

Dall’Alba

et al. 2021

2021 20th International Conference on Advanced Robotics (ICAR)

View full text Add to dashboard Cite

Deep Reinforcement Learning (DRL) is emerging as a promising approach to generate adaptive behaviors for robotic platforms. However, a major drawback of using DRL is the data-hungry training regime that requires millions of trial and error attempts, which is impractical when running experiments on robotic systems. To address this issue, we propose a multi-subtask reinforcement learning method where complex tasks are decomposed manually into low-level subtasks by leveraging human domain knowledge. These subtasks can be parametrized as expert networks and learned via existing DRL methods. Trained subtasks can then be composed with a high-level choreographer. As a testbed, we use a pick and place robotic simulator to demonstrate our methodology, and show that our method outperforms an imitation learning-based method and reaches a high success rate compared to an endto-end learning approach. Moreover, we transfer the learned behavior in a different robotic environment that allows us to exploit sim-to-real transfer and demonstrate the trajectories in a real robotic system. Our training regime is carried out using a central processing unit (CPU)-based system, which demonstrates the data-efficient properties of our approach.

show abstract

Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments

Cited by 3 publications

References 5 publications

Learning Mobile Manipulation

Learning Mobile Manipulation

Divide & Conquer Imitation Learning

Towards Hierarchical Task Decomposition using Deep Reinforcement Learning for Pick and Place Subtasks

Contact Info

Product

Resources

About