Using Deep Reinforcement Learning Methods for Autonomous Vessels in 2D Environments

Etemad, Mohammad; Zare, Nader; Sarvmaili, Mahtab; Soares, Amílcar; Machado, Bruno Brandoli; Matwin, Stan

doi:10.1007/978-3-030-47358-7_21

Cited by 9 publications

(7 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Accordingly, the robot has been set up with four incremental optical rotary shaft encoders, which make wheels odometry calculation possible. The system eventually benefits form this odometry for DC motor control purposes and the localization and autonomous navigation methods [40].…”

Section: Sensorsmentioning

confidence: 99%

Design and implementation of a maxi-sized mobile robot (Karo) for rescue missions

et al. 2021

View full text Add to dashboard Cite

Rescue robots are expected to carry out reconnaissance and dexterity operations in unknown environments comprising unstructured obstacles. Although a wide variety of designs and implementations have been presented within the field of rescue robotics, embedding all mobility, dexterity, and reconnaissance capabilities in a single robot remains a challenging problem. This paper explains the design and implementation of Karo, a mobile robot that exhibits a high degree of mobility at the side of maintaining required dexterity and exploration capabilities for urban search and rescue (USAR) missions. We first elicit the system requirements of a standard rescue robot from the frameworks of Rescue Robot League (RRL) of RoboCup and then, propose the conceptual design of Karo by drafting a locomotion and manipulation system. Considering that, this work presents comprehensive design processes along with detail mechanical design of the robot’s platform and its 7-DOF manipulator. Further, we present the design and implementation of the command and control system by discussing the robot’s power system, sensors, and hardware systems. In conjunction with this, we elucidate the way that Karo’s software system and human–robot interface are implemented and employed. Furthermore, we undertake extensive evaluations of Karo’s field performance to investigate whether the principal objective of this work has been satisfied. We demonstrate that Karo has effectively accomplished assigned standardized rescue operations by evaluating all aspects of its capabilities in both RRL’s test suites and training suites of a fire department. Finally, the comprehensiveness of Karo’s capabilities has been verified by drawing quantitative comparisons between Karo’s performance and other leading robots participating in RRL.

show abstract

Section: Sensorsmentioning

confidence: 99%

Design and implementation of a maxi-sized mobile robot (Karo) for rescue missions

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The environment feature model is represented using the raster method, which divides the environment map into a number of equally sized rasters, with black rasters for obstacles, white rasters for free space, and yellow rasters for target space, each representing a state, so that the complex map environment is divided into feasible and infeasible spaces. The sea environment in which the USV is located is assumed to be a two-dimensional environment [26]. The twodimensional space is discretized into a grid of 8 × 8 raster lengths.…”

Section: Environment Describementioning

confidence: 99%

An Improved Dueling Deep Double-Q Network Based on Prioritized Experience Replay for Path Planning of Unmanned Surface Vehicles

Zhu

et al. 2021

JMSE

View full text Add to dashboard Cite

Unmanned Surface Vehicle (USV) has a broad application prospect and autonomous path planning as its crucial technology has developed into a hot research direction in the field of USV research. This paper proposes an Improved Dueling Deep Double-Q Network Based on Prioritized Experience Replay (IPD3QN) to address the slow and unstable convergence of traditional Deep Q Network (DQN) algorithms in autonomous path planning of USV. Firstly, we use the deep double Q-Network to decouple the selection and calculation of the target Q value action to eliminate overestimation. The prioritized experience replay method is adopted to extract experience samples from the experience replay unit, increase the utilization rate of actual samples, and accelerate the training speed of the neural network. Then, the neural network is optimized by introducing a dueling network structure. Finally, the soft update method is used to improve the stability of the algorithm, and the dynamic ϵ-greedy method is used to find the optimal strategy. The experiments are first conducted in the Open AI Gym test platform to pre-validate the algorithm for two classical control problems: the Cart pole and Mountain Car problems. The impact of algorithm hyperparameters on the model performance is analyzed in detail. The algorithm is then validated in the Maze environment. The comparative analysis of simulation experiments shows that IPD3QN has a significant improvement in learning performance regarding convergence speed and convergence stability compared with DQN, D3QN, PD2QN, PDQN, PD3QN. Also, USV can plan the optimal path according to the actual navigation environment with the IPD3QN algorithm.

show abstract

“…Moreover, many algorithms have been derived by combining the traditional LOS technology and nonlinear control methods [23,24]. In addition, some novelty methods have been applied in the guidance law, such as bioinspired neural [25], deep reinforcement learning methods [26] and vector field [27]. The twin-propeller and twin-hull USV (TPTH-USV) is a usual vehicle for applications due to its good stability and high load [28], such as 'Springer' [29], 'JiuHang-490' [30].…”

Section: Related Workmentioning

confidence: 99%

“…The relationship between the drive voltage and control voltage of the actuator was simply described, v 1 = k 3 •v 2 (26) and the left and right propellers' control voltages for the actuators were:…”

Section: Model Identification For Yaw Motionmentioning

confidence: 99%

A Practical Trajectory Tracking Scheme for a Twin-Propeller Twin-Hull Unmanned Surface Vehicle

Jin

Liu

Wang

et al. 2021

JMSE

View full text Add to dashboard Cite

Trajectory tracking is a basis of motion control for Unmanned Surface Vehicles (USVs), which has been researched well for common USVs. The twin-propeller and twin-hull USV (TPTH-USV) is a special vehicle for applications due to its good stability and high load. We propose a three-layered architecture of trajectory tracking for the TPTH-USV which explicitly decomposes into trajectory guidance, a motion limitator and controller. The trajectory guidance transforms an expected trajectory into an expected speed and expected course in a kinematic layer. The motion limitator describes some restriction for motion features of the USV in the restriction layer, such as the maximum speed and maximum yaw rate. The controller is to control the speed and course of the USV in the kinetic layer. In the first layer, an adaptive line-of-sight guidance law is designed by regulating the speed and course to track a curved line considering the sideslip angle. In the second layer, the motion features are extracted from an identified speed and course coupled model. In the last layer, the course and speed controller are designed based on a twin-PID controller. The feasibility and practicability of the proposed trajectory tracking scheme is validated in sea experiments by a USV called ‘Jiuhang 490’.

show abstract

Using Deep Reinforcement Learning Methods for Autonomous Vessels in 2D Environments

Cited by 9 publications

References 15 publications

Design and implementation of a maxi-sized mobile robot (Karo) for rescue missions

Design and implementation of a maxi-sized mobile robot (Karo) for rescue missions

An Improved Dueling Deep Double-Q Network Based on Prioritized Experience Replay for Path Planning of Unmanned Surface Vehicles

A Practical Trajectory Tracking Scheme for a Twin-Propeller Twin-Hull Unmanned Surface Vehicle

Contact Info

Product

Resources

About