RL-RRT: Kinodynamic Motion Planning via Learning Reachability Estimators From RL Policies

Abstract

This paper addresses two challenges facing sampling-based kinodynamic motion planning: a way to identify good candidate states for local transitions and the subsequent computationally intractable steering between these candidate states. Through the combination of sampling-based planning, a Rapidly Exploring Randomized Tree (RRT) and an efficient kinodynamic motion planner through machine learning, we propose an efficient solution to long-range planning for kinodynamic motion planning. First, we use deep reinforcement learning to learn an obstacle-avoiding policy that maps a robot's sensor observations to actions, which is used as a local planner during planning and as a controller during execution. Second, we train a reachability estimator in a supervised manner, which predicts the RL policy's time to reach a state in the presence of obstacles. Lastly, we introduce RL-RRT that uses the RL policy as a local planner, and the reachability estimator as the distance function to bias tree-growth towards promising regions. We evaluate our method on three kinodynamic systems, including physical robot experiments. Results across all three robots tested indicate that RL-RRT outperforms state of the art kinodynamic planners in efficiency, and also provides a shorter path finish time than a steering function free method. The learned local planner policy and accompanying reachability estimator demonstrate transferability to the previously unseen experimental environments, making RL-RRT fast because the expensive computations are replaced with simple neural network inference. Video: https://youtu.be/dDMVMTOI8KY

Learning to steer a locomotion contact planner

“…In an extension of this work [8], a kinodynamic steering-method is introduced to produce dynamically feasible paths. In a similar spirit to our paper, [43] learns by reinforcement both the local steering method for obstacle avoidance and the control of a wheeled robot, then combines it with a sample-based path planning algorithm to navigate complex environments.…”

Motion Planning for Mobile Robots—Focusing on Deep Reinforcement Learning: A Systematic Review

“…In the aspect of robot motion planning [91] , as shown in FIGURE 11 (a), Andrew [92] used the appropriate "prompt" to shape the optimal return function, and proposed a PEGASUS search strategy to automatically design a stable UAV controller, which got the flight test though the remote control helicopter. [93] added auxiliary tasks of visual data processing to pre train part of the network based on the extended strategy search algorithm, thus significantly reducing the training time and increasing the learning efficiency.…”

Social Robot Navigation Tasks: Combining Machine Learning Techniques and Social Force Model

“…To extend the navigation distance in large indoor environments, AutoRL has been implemented as a local planner for a PRM [46] or RRT [47]. The results in real environments show a high performance and robustness to noise.…”