Unmanned surface vessel obstacle avoidance with prior knowledge‐based reward shaping

Abstract

Autonomous obstacle avoidance control of unmanned surface vessels (USVs) in complex marine environments is always fundamental for its scientific search and detection. Traditional methods usually model USV motion and environments in a mathematical way that needs perceptual information. Unfortunately, it is difficult to provide sufficient perceptual information due to complex marine environments, resulting in inaccurate modeling. Reinforcement learning has recently enjoyed increasing popularity in the problem of obstacle avoidance since it can settle problems by partially observable environment information. However, the autonomous USV obstacle avoidance using reinforcement learning is still facing the challenge of designing appropriate reward functions under complex marine environments. To address these issues, we propose a prior knowledge‐based USV reinforcement learning obstacle avoidance algorithm. In this algorithm, an actor‐critic network is used as the main architecture of the algorithm, and prior knowledge‐based reward shaping used to design relevant reward function for USV obstacle avoidance. A standard USV based on a visual sensor is designed, and the state input of the algorithm is through USV's front vision sensor. We conducted simulation experiments and results prove that our algorithm can effectively converge, and USV achieves high velocity and low collision rate in the complex marine environment.

A Multi-Objective Reinforcement Learning Based Controller for Autonomous Navigation in Challenging Environments

“…Liu et al [14] used virtual potential field to develop a multi-step reinforcement learning algorithm. In [15], deep RL is used for obstacle avoidance in challenging environments. Long et al [16] used a deep neural network in an end-to-end approach for efficient distributed multi-agent navigation, and their algorithm is capable of obstacle avoidance during navigation.…”

“…Many of the works reviewed in the literature either use only two dimensions for the control or do not use the high-dimensional state space of RGB-D as we used. For example, in [15] they focused on obstacle avoidance from RGB image, while our algorithm addresses obstacle avoidance and reaching to goal location from RGB-D image. Moreover, while some works such as [17] focused on following a path, our algorithm generates a path.…”

Reinforcement Learning-Based Autonomous Navigation and Obstacle Avoidance for USVs under Partially Observable Conditions

“…Liu et al [25] presented a dynamic multiple step reinforcement learning algorithm that is based on virtual potential field path planning. Wang et al [26] proposed a deep reinforcement learning method combined with a well-designed reward function to achieve USV obstacle avoidance in complex environments. Long et al [27] proposed an end-to-end obstacle avoidance policy for generating efficient distributed multi-agent navigation, and they gave the expression of the obstacle avoidance navigation policy as a deep neural network mapped from the observed environmental information to the agent's movement speed with steering commands.…”

Geometric relation based point clouds classification and segmentation

“…Generally, the grid-based methods are inevitable to loss 3D spatial information during transformation or voxelization, while most point-based methods ignore the geometric relations in point clouds. Our method further reasons about geometric relations in point clouds, as prior knowledge, 25,26 which is crucial to improve the classification and segmentation performance.…”