Adaptive and extendable control of unmanned surface vehicle formations using distributed deep reinforcement learning

Wang, Shuwu; Ma, Feng; Yan, Xinping; Wu, Peng; Liu, Yuanchang

doi:10.1016/j.apor.2021.102590

Cited by 40 publications

(8 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The leader-follower method is widely applied to design formation control for USVs [25,26]. Wang et al [27] proposed a distributed DRL algorithm for USV formations, which is capable to arbitrarily increase the number of ships or change formation shapes. The individual intelligence in swarm dynamics is simple.…”

Section: Literature Review 21 Ship Collision Avoidance Methodsmentioning

confidence: 99%

A Novel Ship Collision Avoidance Awareness Approach for Cooperating Ships Using Multi-Agent Deep Reinforcement Learning

Chen

et al. 2021

JMSE

Self Cite

View full text Add to dashboard Cite

Ships are special machineries with large inertias and relatively weak driving forces. Simulating the manual operations of manipulating ships with artificial intelligence (AI) and machine learning techniques becomes more and more common, in which avoiding collisions in crowded waters may be the most challenging task. This research proposes a cooperative collision avoidance approach for multiple ships using a multi-agent deep reinforcement learning (MADRL) algorithm. Specifically, each ship is modeled as an individual agent, controlled by a Deep Q-Network (DQN) method and described by a dedicated ship motion model. Each agent observes the state of itself and other ships as well as the surrounding environment. Then, agents analyze the navigation situation and make motion decisions accordingly. In particular, specific reward function schemas are designed to simulate the degree of cooperation among agents. According to the International Regulations for Preventing Collisions at Sea (COLREGs), three typical scenarios of simulation, which are head-on, overtaking and crossing, are established to validate the proposed approach. With sufficient training of MADRL, the ship agents were capable of avoiding collisions through cooperation in narrow crowded waters. This method provides new insights for bionic modeling of ship operations, which is of important theoretical and practical significance.

show abstract

Section: Literature Review 21 Ship Collision Avoidance Methodsmentioning

confidence: 99%

A Novel Ship Collision Avoidance Awareness Approach for Cooperating Ships Using Multi-Agent Deep Reinforcement Learning

Chen

et al. 2021

JMSE

Self Cite

View full text Add to dashboard Cite

show abstract

“…With the development of artificial intelligence algorithms, applied reinforcement learning (RL) has been introduced to path planning and navigation. Wang et al [29] proposed a distributed deep reinforcement learning (DRL) algorithm for unmanned surface vehicle (USV) formations based on the learning of two key abilities, adaptability and scalability, such that the formation can arbitrarily increase the number of USVs or change the formation's shape. A path-planning strategy based on DRL and collision-avoidance functions was found to solve path-planning problems associated with unmanned craft in uncertain environments [30].…”

Section: Introductionmentioning

confidence: 99%

Adaptive Sampling Path Planning for a 3D Marine Observation Platform Based on Evolutionary Deep Reinforcement Learning

Zhang,

Liu,

Zhou

2023

JMSE

View full text Add to dashboard Cite

Adaptive sampling of the marine environment may improve the accuracy of marine numerical prediction models. This study considered adaptive sampling path optimization for a three-dimensional (3D) marine observation platform, leading to a path-planning strategy based on evolutionary deep reinforcement learning. The low sampling efficiency of the reinforcement learning algorithm is improved by evolutionary learning. The combination of these two components as a new algorithm has become a current research trend. We first combined the evolutionary algorithm with different reinforcement learning algorithms to verify the effectiveness of the combination of algorithms with different strategies. Experimental results indicate that the fusion of the two algorithms based on a maximum-entropy strategy is more effective for adaptive sampling using a 3D marine observation platform. Data assimilation experiments indicate that adaptive sampling data from a 3D mobile observation platform based on evolutionary deep reinforcement learning improves the accuracy of marine environment numerical prediction systems.

show abstract

“…Similarly, the leader-follower method, LOS strategy and neural networks were used in [18], for designing the formation controller of a waterjet USV, exposed to unmodelled dynamics, environmental disturbances, input saturation, and output constraints. A USV formation approach, based on a distributed deep reinforcement learning algorithm, was proposed in [19], which made formations to arbitrarily increase the number of USVs or change formation forms. In [20], model predictive control was used to deal with vessel train formation (VTF) problems including cooperative collision avoidance and grouping of vessels; a single-layer serial iterative architecture was adopted in distributed formulation, for reducing communication requirements and improving robustness against failures.…”

Section: Introductionmentioning

confidence: 99%

An Improved Dynamic Surface Sliding Mode Method for Autonomous Cooperative Formation Control of Underactuated USVS with Complex Marine Environment Disturbances

Dong

et al. 2022

Polish Maritime Research

View full text Add to dashboard Cite

In this paper, a novel dynamic surface sliding mode control (DSSMC) method, combined with a lateral velocity tracking differentiator (LVTD), is proposed for the cooperative formation control of underactuated unmanned surface vehicles (USVs) exposed to complex marine environment disturbances. Firstly, in view of the kinematic and dynamic models of USVs and the design idea of a virtual control law in a backstepping approach, the trajectory tracking control problem of USVs’ cooperative formation is transformed into a stabilisation problem of the virtual control law of longitudinal and lateral velocities. Then, aiming at the problem of differential explosion caused by repeated derivation in the process of backstepping design, the first-order low-pass filter about the virtual longitudinal velocity and intermediate state quantity of position is constructed to replace differential calculations during the design of the control law, respectively. In order to reduce the steady-state error when stabilising the virtual lateral velocity control law, the integral term is introduced into the design of the sliding mode surface with a lateral velocity error, and then the second-order sliding mode surface with an integral is structured. In addition, due to the problem of controller oscillation and the role of the tracking differentiator (TD) in active disturbance rejection control (ADRC), the LVTD is designed to smooth the state quantity of lateral velocity. Subsequently, based on the dynamic model of USV under complex marine environment disturbances, the nonlinear disturbance observer is designed to observe the disturbances and compensate the control law. Finally, the whole cooperative formation system is proved to be uniformly and ultimately bounded, according to the Lyapunov stability theory, and the stability and validity of the method is also verified by the simulation results.

show abstract

Adaptive and extendable control of unmanned surface vehicle formations using distributed deep reinforcement learning

Cited by 40 publications

References 24 publications

A Novel Ship Collision Avoidance Awareness Approach for Cooperating Ships Using Multi-Agent Deep Reinforcement Learning

A Novel Ship Collision Avoidance Awareness Approach for Cooperating Ships Using Multi-Agent Deep Reinforcement Learning

Adaptive Sampling Path Planning for a 3D Marine Observation Platform Based on Evolutionary Deep Reinforcement Learning

An Improved Dynamic Surface Sliding Mode Method for Autonomous Cooperative Formation Control of Underactuated USVS with Complex Marine Environment Disturbances

Contact Info

Product

Resources

About