Navigation in Unknown Dynamic Environments Based on Deep Reinforcement Learning

Abstract

In this paper, we propose a novel Deep Reinforcement Learning (DRL) algorithm which can navigate non-holonomic robots with continuous control in an unknown dynamic environment with moving obstacles. We call the approach MK-A3C (Memory and Knowledge-based Asynchronous Advantage Actor-Critic) for short. As its first component, MK-A3C builds a GRU-based memory neural network to enhance the robot’s capability for temporal reasoning. Robots without it tend to suffer from a lack of rationality in face of incomplete and noisy estimations for complex environments. Additionally, robots with certain memory ability endowed by MK-A3C can avoid local minima traps by estimating the environmental model. Secondly, MK-A3C combines the domain knowledge-based reward function and the transfer learning-based training task architecture, which can solve the non-convergence policies problems caused by sparse reward. These improvements of MK-A3C can efficiently navigate robots in unknown dynamic environments, and satisfy kinetic constraints while handling moving objects. Simulation experiments show that compared with existing methods, MK-A3C can realize successful robotic navigation in unknown and challenging environments by outputting continuous acceleration commands.

Deep reinforcement learning based mobile robot navigation: A review

“…As a variant of LSTM, the Gated Recurrent Unit (GRU) is easier to train. Zeng et al [80] built a GRU-based memory neural network and proved that it could improve performance in complex navigation tasks.…”

Motion Planning for Mobile Robots—Focusing on Deep Reinforcement Learning: A Systematic Review

“…Zeng et al [113] proposed an Actor-Critic method (MK-A3C) based on asynchronous advantages of memory and knowledge for the navigation problem of nonholonomic robots with continuous control. MK-A3C constructs a memory neural network based on GRU to enhance the temporal reasoning ability of the robot.At the same time, the robot is given a certain memory ability.…”

An Extensive Review of Mobile Agricultural Robotics for Field Operations: Focus on Cotton Harvesting

“…There are several path planning algorithms developed for robotics systems, such as grid-based search algorithms (assumes every point/object is covered in a grid configuration [44,45]), interval-based search algorithms (generates paving to cover an entire configuration space instead of grid [44]), geometric algorithms (find safe path from the start to goal initially [46]), reward-based algorithms (a robot tries to take a path, and it is rewarded positively if successful and negatively if otherwise [47]), artificial potential fields algorithms (robot is modeled to be attracted to positive path and repelled by obstacles [48]), and sampling-based algorithms (path is found from the roadmap spaces of the configuration space). Each of the algorithms has potential use, and some are just classic methods like grid-based algorithms [49].…”