Adaptive low-level control of autonomous underwater vehicles using deep reinforcement learning

Carlucho, Ignacio; Paula, Mariano De; Wang, Sen; Pétillot, Yvan; Acosta, Gerardo

doi:10.1016/j.robot.2018.05.016

Cited by 142 publications

(58 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The reward function is one of the major ways the designer can influence and direct the behaviour of the agent. One of the more popular alternatives to L 1 norm and clipping to achieve saturated rewards are the class of exponential reward functions, and notably the Gaussian reward function as in [37]. Analyzing different choices of the reward function was not given much focus as the original choice gave satisfying results.…”

Section: A Key Factors Impacting Trainingmentioning

confidence: 99%

Deep Reinforcement Learning Attitude Control of Fixed-Wing UAVs Using Proximal Policy optimization

Bøhn

Coates

Moe

et al. 2019

2019 International Conference on Unmanned Aircraft Systems (ICUAS)

152

View full text Add to dashboard Cite

Contemporary autopilot systems for unmanned aerial vehicles (UAVs) are far more limited in their flight envelope as compared to experienced human pilots, thereby restricting the conditions UAVs can operate in and the types of missions they can accomplish autonomously. This paper proposes a deep reinforcement learning (DRL) controller to handle the nonlinear attitude control problem, enabling extended flight envelopes for fixed-wing UAVs. A proof-of-concept controller using the proximal policy optimization (PPO) algorithm is developed, and is shown to be capable of stabilizing a fixed-wing UAV from a large set of initial conditions to reference roll, pitch and airspeed values. The training process is outlined and key factors for its progression rate are considered, with the most important factor found to be limiting the number of variables in the observation vector, and including values for several previous time steps for these variables. The trained reinforcement learning (RL) controller is compared to a proportional-integralderivative (PID) controller, and is found to converge in more cases than the PID controller, with comparable performance. Furthermore, the RL controller is shown to generalize well to unseen disturbances in the form of wind and turbulence, even in severe disturbance conditions.

show abstract

Section: A Key Factors Impacting Trainingmentioning

confidence: 99%

Deep Reinforcement Learning Attitude Control of Fixed-Wing UAVs Using Proximal Policy optimization

Bøhn

Coates

Moe

et al. 2019

2019 International Conference on Unmanned Aircraft Systems (ICUAS)

152

View full text Add to dashboard Cite

show abstract

“…Waterborne transport vehicles are divided into surface water ships and underwater ships. For underwater ships, relevant scholars use DRL based on actor-critic to achieve low-level control of autonomous underwater vehicles (AUV) [231]. For example, two neural networks are used to construct DRL [232], then actor-critic [233] or deep deterministic policy gradient (DDPG) is introduced to self-modify in subsequent research [234].…”

Section: Artificial Intelligencementioning

confidence: 99%

State-of-the-Art Research on Motion Control of Maritime Autonomous Surface Ships

Wang

Liu

et al. 2019

JMSE

View full text Add to dashboard Cite

At present, with the development of waterborne transport vehicles, research on ship faces a new round of challenges in terms of intelligence and autonomy. The concept of maritime autonomous surface ships (MASS) has been put forward by the International Maritime Organization in 2017, in which MASS become the new focus of the waterborne transportation industry. This paper elaborates on the state-of-the-art research on motion control of MASS. Firstly, the characteristics and current research status of unmanned surface vessels in MASS and conventional ships are summarized, and the system composition of MASS is analyzed. In order to better realize the self-adaptability of the MASS motion control, the theory and algorithm of ship motion control-related systems are emphatically analyzed under the condition of classifying ship motion control. Especially, the application of intelligent algorithms in the ship control field is summarized and analyzed. Finally, this paper summarizes the challenges faced by MASS in the model establishment, motion control algorithms, and real ship experiments, and proposes the composition of MASS motion control system based on variable autonomous control strategy. Future researches on the accuracy and diversity of developments and applications to MASS motion control are suggested.

show abstract

“…The goal in reinforcement learning is to learn a policy that maximizes the expected return J from the start distribution [9]:…”

Section: Deep Reinforcement Learning Theorymentioning

confidence: 99%

“…His method was easier to implement compared with other RL methods, but at the same time, simulated results showed a poor speed of convergence towards the minimal solution [8]. Carlucho et al developed a deep RL framework based on the Actor-Critic theory, which took the available raw sensory information as input and output the continuous control actions as low-level control commands of AUV [9]. In another article [10], an expert agent-based system, based on a reinforcement learning agent, was proposed for self-adapting multiple low-level PID controllers in mobile robots.…”

Section: Introductionmentioning

confidence: 99%

Three-Dimensional Path Tracking Control of Autonomous Underwater Vehicle Based on Deep Reinforcement Learning

Sun

Zhang

et al. 2019

JMSE

View full text Add to dashboard Cite

In this paper, the three-dimensional (3D) path tracking control of an autonomous underwater vehicle (AUV) under the action of sea currents was researched. A novel reward function was proposed to improve learning ability and a disturbance observer was developed to observe the disturbance caused by currents. Based on existing models, the dynamic and kinematic models of the AUV were established. Deep Deterministic Policy Gradient, a deep reinforcement learning, was employed for designing the path tracking controller. Compared with the backstepping sliding mode controller, the controller proposed in this article showed excellent performance, at least in the particular study developed in this article. The improved reward function and the disturbance observer were also found to work well with improving path tracking performance.

show abstract

Adaptive low-level control of autonomous underwater vehicles using deep reinforcement learning

Abstract: low-level control of autonomous underwater vehicles using deep reinforcement learning, Robotics and Autonomous Systems (2018),

Cited by 142 publications

References 53 publications

Deep Reinforcement Learning Attitude Control of Fixed-Wing UAVs Using Proximal Policy optimization

Deep Reinforcement Learning Attitude Control of Fixed-Wing UAVs Using Proximal Policy optimization

State-of-the-Art Research on Motion Control of Maritime Autonomous Surface Ships

Three-Dimensional Path Tracking Control of Autonomous Underwater Vehicle Based on Deep Reinforcement Learning

Contact Info

Product

Resources

About