Deep Reinforcement Learning for Vectored Thruster Autonomous Underwater Vehicle Control

Abstract

Autonomous underwater vehicles (AUVs) are widely used to accomplish various missions in the complex marine environment; the design of a control system for AUVs is particularly difficult due to the high nonlinearity, variations in hydrodynamic coefficients, and external force from ocean currents. In this paper, we propose a controller based on deep reinforcement learning (DRL) in a simulation environment for studying the control performance of the vectored thruster AUV. RL is an important method of artificial intelligence that can learn behavior through trial-and-error interactions with the environment, so it does not need to provide an accurate AUV control model that is very hard to establish. The proposed RL algorithm only uses the information that can be measured by sensors inside the AUVs as the input parameters, and the outputs of the designed controller are the continuous control actions, which are the commands that are set to the vectored thruster. Moreover, a reward function is developed for deep RL controller considering different factors which actually affect the control accuracy of AUV navigation control. To confirm the algorithm’s effectiveness, a series of simulations are carried out in the designed simulation environment, which is a method to save time and improve efficiency. Simulation results prove the feasibility of the deep RL algorithm applied to the control system for AUV. Furthermore, our work also provides an optional method for robot control problems to deal with improving technology requirements and complicated application environments.

Deep Reinforcement Learning Based Three-dimensional Path Tracking Control of An Underwater Robot

“…The dynamic control of the robot adopts the deep RL method [9]. First, the task needs to be modeled as a Markov decision process which means that the state at the current time is only related to the state and action at the previous time.…”

Three-Dimensional Path-Following Control of an Autonomous Underwater Vehicle Based on Deep Reinforcement Learning

“…By using the RL-based control [23], the desired dynamic control laws of the vehicle, regardless of the dynamic model, can be obtained, where the interaction between the vehicle and the environment is performed. This control process can be described as a Markov decision process, where the AUV performs actions under the current states and behaviour policy.…”

“…In the TD3 network design, the structure of the actor network and critic network is shown in Fig. 5, which has fewer nodes than the previous work [23]. The inputs of the actor network are states and the outputs are actions.…”

AUV Trajectory Tracking Models and Control Strategies: A Review

“…Considering different factors which actually affect the control accuracy of AUV navigation control, ref. [133] developed a reward function for deep RL controller. The designed reward function can effectively improve reliability and stability, reduce energy consumption, and restrain the vectored thruster sudden change.…”