Deep Reinforcement Learning for Mapless Navigation of a Hybrid Aerial Underwater Vehicle with Medium Transition

Grando, Ricardo Bedin; Jesus, Junior Costa de; Kich, Victor Augusto; Kolling, Alisson Henrique; Bortoluzzi, Nicolas Pieper; Pinheiro, Pedro M.; Neto, Armando Alves; Drews, Paul

doi:10.1109/icra48506.2021.9561188

Cited by 37 publications

(31 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PPO is a classic online RL algorithm that can be used to solve the sequential decision-making problem of unmanned aerial vehicles and non-linear attitude control problems [40,48]. SAC is a representative algorithm of offline RL that can realize low-level control of quadrotors and map-free navigation and obstacle avoidance of hybrid unmanned underwater vehicles [49,50]. GAIL is a representative IL algorithm that predicts airport-airside motion of aircraft-taxi trajectories and enables mobile robots to learn to navigate in dynamic pedestrian environments in a socially desirable manner [51,52].…”

Section: Resultsmentioning

confidence: 99%

An AUV Target-Tracking Method Combining Imitation Learning and Deep Reinforcement Learning

Mao

Gao

Zhang

et al. 2022

JMSE

View full text Add to dashboard Cite

This study aims to solve the problem of sparse reward and local convergence when using a reinforcement learning algorithm as the controller of an AUV. Based on the generative adversarial imitation (GAIL) algorithm combined with a multi-agent, a multi-agent GAIL (MAG) algorithm is proposed. The GAIL enables the AUV to directly learn from expert demonstrations, overcoming the difficulty of slow initial training of the network. Parallel training of multi-agents reduces the high correlation between samples to avoid local convergence. In addition, a reward function is designed to help training. Finally, the results show that in the unity simulation platform test, the proposed algorithm has a strong optimal decision-making ability in the tracking process.

show abstract

Section: Resultsmentioning

confidence: 99%

An AUV Target-Tracking Method Combining Imitation Learning and Deep Reinforcement Learning

Mao

Gao

Zhang

et al. 2022

JMSE

View full text Add to dashboard Cite

show abstract

“…The network architecture used by the Deep-RL resembles the SAC network presented in the works of de Jesus et al [24], and Grando et al [25]. Fig.…”

Section: Methodsmentioning

confidence: 99%

Depth-CUPRL: Depth-Imaged Contrastive Unsupervised Prioritized Representations in Reinforcement Learning for Mapless Navigation of Unmanned Aerial Vehicles

Jesus¹,

Kich²,

Kolling³

et al. 2022

Preprint

View full text Add to dashboard Cite

Reinforcement Learning (RL) has presented an impressive performance in video games through raw pixel imaging and continuous control tasks. However, RL performs poorly with high-dimensional observations such as raw pixel images. It is generally accepted that physical state-based RL policies such as laser sensor measurements give a more sampleefficient result than learning by pixels. This work presents a new approach that extracts information from a depth map estimation to teach an RL agent to perform the mapless navigation of Unmanned Aerial Vehicle (UAV). We propose the Depth-Imaged Contrastive Unsupervised Prioritized Representations in Reinforcement Learning(Depth-CUPRL) that estimates the depth of images with a prioritized replay memory. We used a combination of RL and Contrastive Learning to lead with the problem of RL based on images. From the analysis of the results with Unmanned Aerial Vehicles (UAVs), it is possible to conclude that our Depth-CUPRL approach is effective for the decision-making and outperforms state-of-the-art pixel-based approaches in the mapless navigation capability.

show abstract

“…From the first works regarding HUAUVs [3,10,11,12] to recent studies [5,8,13], no major contributions were added to the high-level motion planning of quadrotor-based HUAUVs branch. In recent work, Grando et al [2] presented a navigation strategy based on reinforcement learning to HUAUVs. Nevertheless, their method is limited to pre-trained cluttered environments.…”

Section: Related Workmentioning

confidence: 99%

“…Vehicles capable of acting both in the air and under the water offer a large number of applications in a variety of scenarios, some of them extreme, ranging from the extreme cold of polar regions to the hot and humid climate of rain forests. Operating and transiting between air and water in such challenging environments are difficult tasks, with many setbacks to achieve a good perception and actuation overall [2]. Most of these Hybrid Unmanned Aerial Underwater Vehicles (HUAUVs) were inspired by aerial vehicles, such as quadcopters and hexacopters [3].…”

Section: Introductionmentioning

confidence: 99%

Trajectory Planning for Hybrid Unmanned Aerial Underwater Vehicles with Smooth Media Transition

Pinheiro¹,

Neto²,

Grando³

et al. 2021

Preprint

View full text Add to dashboard Cite

In the last decade, a great effort has been employed in the study of Hybrid Unmanned Aerial Underwater Vehicles, robots that can easily fly and dive into the water with different levels of mechanical adaptation. However, most of this literature is concentrated on physical design, practical issues of construction, and, more recently, low-level control strategies. Little has been done in the context of high-level intelligence, such as motion planning and interactions with the real world. Therefore, we proposed in this paper a trajectory planning approach that allows collision avoidance against unknown obstacles and smooth transitions between aerial and aquatic media. Our method is based on a variant of the classic Rapidly-exploring Random Tree, whose main advantages are the capability to deal with obstacles, complex nonlinear dynamics, model uncertainties, and external disturbances. The approach uses the dynamic model of the HyDrone, a hybrid vehicle proposed with high underwater performance, but we believe it can be easily generalized to other types of aerial/aquatic platforms. In the experimental section, we present simulated results in environments filled with obstacles, where the robot is commanded to perform different media movements, demonstrating the applicability of our strategy.

show abstract

Deep Reinforcement Learning for Mapless Navigation of a Hybrid Aerial Underwater Vehicle with Medium Transition

Cited by 37 publications

References 28 publications

An AUV Target-Tracking Method Combining Imitation Learning and Deep Reinforcement Learning

An AUV Target-Tracking Method Combining Imitation Learning and Deep Reinforcement Learning

Depth-CUPRL: Depth-Imaged Contrastive Unsupervised Prioritized Representations in Reinforcement Learning for Mapless Navigation of Unmanned Aerial Vehicles

Trajectory Planning for Hybrid Unmanned Aerial Underwater Vehicles with Smooth Media Transition

Contact Info

Product

Resources

About