A Novel Ship Collision Avoidance Awareness Approach for Cooperating Ships Using Multi-Agent Deep Reinforcement Learning

Abstract

Ships are special machineries with large inertias and relatively weak driving forces. Simulating the manual operations of manipulating ships with artificial intelligence (AI) and machine learning techniques becomes more and more common, in which avoiding collisions in crowded waters may be the most challenging task. This research proposes a cooperative collision avoidance approach for multiple ships using a multi-agent deep reinforcement learning (MADRL) algorithm. Specifically, each ship is modeled as an individual agent, controlled by a Deep Q-Network (DQN) method and described by a dedicated ship motion model. Each agent observes the state of itself and other ships as well as the surrounding environment. Then, agents analyze the navigation situation and make motion decisions accordingly. In particular, specific reward function schemas are designed to simulate the degree of cooperation among agents. According to the International Regulations for Preventing Collisions at Sea (COLREGs), three typical scenarios of simulation, which are head-on, overtaking and crossing, are established to validate the proposed approach. With sufficient training of MADRL, the ship agents were capable of avoiding collisions through cooperation in narrow crowded waters. This method provides new insights for bionic modeling of ship operations, which is of important theoretical and practical significance.

Priority Setting and Resource Allocation in Coastal Local Government Marine Regulatory Reform: Application of Machine Learning in Resource Optimization

“…For instance, Glaviano et al [25] introduced a marine supervision model grounded in deep learning, facilitating real-time monitoring and early warning systems for the marine environment, thus offering innovative approaches to marine environmental protection and management. Additionally, Chen et al [26] proposed an ocean supervision strategy based on reinforcement learning, fostering automation and intelligence in supervision tasks through the training of intelligent agents to learn optimal supervision strategies. Wang et al [27] introduced an innovative approach to optimize the blockchain network system by employing smart contracts to construct a risk management system for online public opinion [27].…”

A Multi-Ship Collision Avoidance Algorithm Using Data-Driven Multi-Agent Deep Reinforcement Learning

“…Chen et al proposed a multi-ship cooperative collision avoidance method based on the MADRL algorithm. By designing different reward weights to vary the degree of cooperation among the agents, the impact of agents in different cooperation modes on their collision avoidance behavior is discussed [20]. However, the above DRL algorithms are constructed and trained by pure simulation data.…”

“…Therefore, this paper will adopt the second avoidance measure as the action space, through a series of discrete course angle commands to continuously adjust the course and finally complete the ship collision avoidance. In other words, the discrete course change angle range is set as this algorithm's action space [20].…”

Multiple Container Terminal Berth Allocation and Joint Operation Based on Dueling Double Deep Q-Network

“…Instead, it learns decision policies adaptively through continuous interaction with the environment, achieving optimal or near-optimal solutions. This characteristic endows RL with clear superiority when dealing with dynamism, uncertainty, and numerous complex constraints [37][38][39]. For instance, when confronted with resource-constrained scheduling issues, reinforcement learning possesses online learning and adaptive capabilities, allowing it to flexibly balance potentially conflicting objectives and formulate efficient scheduling strategies while adhering to operational constraints.…”