A Multi-Ship Collision Avoidance Algorithm Using Data-Driven Multi-Agent Deep Reinforcement Learning

Abstract

Maritime Autonomous Surface Ships (MASS) are becoming of interest to the maritime sector and are also on the agenda of the International Maritime Organization (IMO). With the boom in global maritime traffic, the number of ships is increasing rapidly. The use of intelligent technology to achieve autonomous collision avoidance is a hot issue widely discussed in the industry. In the endeavor to solve this problem, multi-ship coordinated collision avoidance has become a crucial challenge. This paper proposes a multi-ship autonomous collision avoidance decision-making algorithm by a data-driven method and adopts the Multi-agent Deep Reinforcement Learning (MADRL) framework for its design. Firstly, the overall framework of this paper and its components follow the principle of “reality as primary and simulation as supplementary”, so a real data-driven AIS (Automatic Identification System) dominates the model construction. Secondly, the agent’s observation state is determined by quantifying the hazardous area. Then, based on a full understanding of the International Regulations for Preventing Collisions at Sea (COLREGs) and the preliminary data collection, this paper combines the statistical results of the real water traffic data to guide and design the algorithm framework and selects the representative influencing factors to be designed in the collision avoidance decision-making algorithm’s reward function. Next, we train the algorithmic model using both real data and simulation data. Meanwhile, Prioritized Experience Replay (PER) is adopted to accelerate the model’s learning efficiency. Finally, 40 encounter scenarios are designed and extended to verify the algorithm performance based on the idea of the Imazu problem. The experimental results show that this algorithm can efficiently make a ship collision avoidance decision in compliance with COLREGs. Multi-agent learning through shared network policies can ensure that the agents pass beyond the safe distance in unknown environments. We can apply the trained model to the system with different numbers of agents to provide a reference for the research of autonomous collision avoidance in ships.

Collision-Free Formation-Containment Tracking of Multi-USV Systems with Constrained Velocity and Driving Force

“…In recent years, with the continuous development of unmanned surface vehicle (USV) swarm coordination technology, the formation technology of multiple USVs has been increasingly applied in fields such as marine data collection [1], collaborative search and rescue [2,3], cooperative escorting [4], and collaborative transportation [5]. During the execution of various formation tasks by multiple USV clusters, collision avoidance among USVs, as well as avoidance of obstacles such as reefs, buoys, and ice floes on the sea surface, is a fundamental requirement [6][7][8][9][10]. Currently, substantial research has focused on collision avoidance within multi-USV formations [8,[11][12][13][14].…”

Identification of Multi-Innovation Stochastic Gradients with Maximum Likelihood Algorithm Based on Ship Maneuverability and Wave Peak Models

“…As a crucial industry for economic development, the shipping industry is responsible for transporting over 90% of global trade cargo [1,2]. The performance of ship motion control systems directly impacts shipping safety and economic costs.…”

Maritime Autonomous Surface Ships

“…Firstly, one review paper on developing Digital Twin (DT) technology in the maritime domain is provided [1]. The following topics on autonomous surface ships are included in this book: methods of ship control [2][3][4][5], collision avoidance [6,7], ship detection methods [8,9], and manoeuvring models [10].…”

“…Another important topic for autonomous surface ships is collision avoidance, since the ship must be able to avoid unexpected obstacles. Niu et al [6] proposed a multi-ship autonomous collision avoidance decision-making algorithm using a data-driven method and adopted the Multi-agent Deep Reinforcement Learning (MADRL) framework. The 40 encounter scenarios were designed to verify the proposed algorithm, and the results show that this algorithm can efficiently make a ship collision avoidance decision in compliance with COLREGs.…”