Learning Navigation Behaviors End-to-End With AutoRL

Abstract

We learn end-to-end point-to-point and pathfollowing navigation behaviors that avoid moving obstacles. These policies receive noisy lidar observations and output robot linear and angular velocities. The policies are trained in small, static environments with AutoRL, an evolutionary automation layer around Reinforcement Learning (RL) that searches for a deep RL reward and neural network architecture with large-scale hyper-parameter optimization. AutoRL first finds a reward that maximizes task completion, and then finds a neural network architecture that maximizes the cumulative of the found reward. Empirical evaluations, both in simulation and on-robot, show that AutoRL policies do not suffer from the catastrophic forgetfulness that plagues many other deep reinforcement learning algorithms, generalize to new environments and moving obstacles, are robust to sensor, actuator, and localization noise, and can serve as robust building blocks for larger navigation tasks. Our path-following and point-topoint policies are respectively 23% and 26% more successful than comparison methods across new environments. Video at: https://youtu.be/0UwkjpUEcbI.

RL-RRT: Kinodynamic Motion Planning via Learning Reachability Estimators From RL Policies

“…It has three main steps. First, we learn an obstacle-avoiding point to point policy with AutoRL [4]. Next, since the RL policy avoids obstacles, we can use the policy to generate obstacle-aware reachability training samples by repeatedly rolling out the learned policy.…”

“…Here we show that this is not the case when proxy rewards are used. AutoRL uses proxy rewards (shown in Section III-A) since they significantly improve learning performance, especially for tasks with sparse learning sigals such as navigation [4]. Fig 6a shows examples of two Asteroid trajectories and Fig.…”

“…Recently, deep Reinforcement Learning (RL) emerged as a promising near optimal steering function for kinodynamic systems [12]. In addition, deep RL algorithms can learn policies that map noisy lidar or camera observations directly to robot actions, thus enabling obstacle avoidance while navigating between states for differential drive robots [4], [5]. With the recent development of AutoRL [4], which uses evolutionary algorithms to largely eliminate the need to handtune hyper-parameters, network structure and reward functions.…”

“…In addition, deep RL algorithms can learn policies that map noisy lidar or camera observations directly to robot actions, thus enabling obstacle avoidance while navigating between states for differential drive robots [4], [5]. With the recent development of AutoRL [4], which uses evolutionary algorithms to largely eliminate the need to handtune hyper-parameters, network structure and reward functions. This combination offers the promise of deep RL being employed for local planning, i.e., providing both steering function and obstacle avoidance.…”

“…This is a mapless, goal-conditioned policy that maps sensor readings to control. These P2P policies generalize to new environments without re-training [4]. Second, we train a supervised obstacle-aware reachability estimator that predicts the time it takes the P2P policy to guide the robot from a start to goal state in presence of obstacles, using local observations such as lidar.…”