Learn to Navigate: Cooperative Path Planning for Unmanned Surface Vehicles Using Deep Reinforcement Learning

Abstract

Unmanned surface vehicle (USV) has witnessed a rapid growth in the recent decade and has been applied in various practical applications in both military and civilian domains. USVs can either be deployed as a single unit or multiple vehicles in a fleet to conduct ocean missions. Central to the control of USV and USV formations, path planning is the key technology that ensures the navigation safety by generating collision free trajectories. Compared with conventional path planning algorithms, the deep reinforcement learning (RL) based planning algorithms provides a new resolution by integrating a high-level artificial intelligence. This work investigates the application of deep reinforcement learning algorithms for USV and USV formation path planning with specific focus on a reliable obstacle avoidance in constrained maritime environments. For single USV planning, with the primary aim being to calculate a shortest collision avoiding path, the designed RL path planning algorithm is able to solve other complex issues such as the compliance with vehicle motion constraints. The USV formation maintenance algorithm is capable of calculating suitable paths for the formation and retain the formation shape robustly or vary shapes where necessary, which is promising to assist with the navigation in environments with cluttered obstacles. The developed three sets of algorithms are validated and tested in computer-based simulations and practical maritime environments extracted from real harbour areas in the UK. INDEX TERMS Deep reinforcement learning, motion planning, multi-agent systems, unmanned surface vehicles (USVs), USV formations.

A Multiagent Deep Reinforcement Learning Approach for Path Planning in Autonomous Surface Vehicles: The Ypacaraí Lake Patrolling Case

“…However, there are several challenges to be addressed [7], like multiagent credit assignment, global exploration, relative over-generalization, and scalability. It is remarkable, in the context of optimization of ASVs fleets, the contributions of [17], where a fleet meta-agent of three boat-like autonomous vehicles is trained using Deep Q-Learning (DQL) to perform swarmcooperative trajectories. The multi agent local trajectory optimization is addressed also in [18], where the DRL goal is to optimize the policies of 3-5 agents to reach several final positions trough static obstacles.…”

“…This dimension problem becomes unfeasible with a large number of agents, limiting the scalability of such methods as they are unable to deal with changes in the fleet size. This is the case in [17], where the fleet size is fixed and the action space is small (|A | = 27) and more vehicles will explode the scale of the problem. Some researches try to deal with the drawbacks of both methodologies by designing combined methodologies between the pure independent approach and a centralized learning like [14], [15].…”

Motion Planning for Mobile Robots—Focusing on Deep Reinforcement Learning: A Systematic Review

“…The working mode of multi-robot cooperation brings more challenges to the inter-individual motion planning in the group. How to carry out the cooperative motion planning effectively becomes the unique feature of this field, which is different from the single robot motion planning.The architecture of reinforcement learning motion planning system for multimobile robots can be mainly divided into two categories: centralized [129] and distributed [130] . Centralized reinforcement learning takes the common task of multiple robots as the training goal, and there is a centralized computing unit that can obtain the state and sensor information of all robots, and the centralized computing unit is responsible for the centralized strategy training and distribution.…”

Reinforcement Learning for Position Control Problem of a Mobile Robot

“…Stability of the closed loop system was achieved by the use of an additional supervisory line in the control law. Finally, in [24], the authors present the application of deep reinforcement learning algorithms for mobile robots and formation path planning with a specific focus on reliable obstacle avoidance in constrained maritime environments. The designed RL path planning algorithm is able to solve other complex issues such as the compliance with vehicle motion constraints.…”