Obstacle Avoidance for UAS in Continuous Action Space Using Deep Reinforcement Learning

Abstract

Obstacle avoidance for small unmanned aircraft is vital for the safety of future urban air mobility (UAM) and Unmanned Aircraft System (UAS) Traffic Management (UTM). There are a variety of techniques for real-time robust drone guidance, but numerous of them solve in discretized airspace and control, which would require an additional path smoothing step to provide flexible commands for UAS. To deliver safe and computationally efficient guidance for UAS operations, we explore the use of a deep reinforcement learning algorithm based on Proximal Policy Optimization (PPO) to lead autonomous UAS to their destinations while bypassing obstacles through continuous control. The proposed scenario state representation and reward function can map the continuous state space to continuous control for both heading angle and speed. To verify the effectiveness of the proposed learning framework, we conducted numerical experiments with static and moving obstacles. Uncertainties associated with the environments and safety operation bounds are investigated in detail. Results show that the proposed model is able to provide accurate and robust guidance and resolve conflict with a success rate of over 99%.INDEX TERMS continuous control, deep reinforcement learning, UAS obstacle avoidance, uncertainty.

Adaptive Collision Avoidance for Multiple UAVs in Urban Environments

“…In [25], a proper heading angle was obtained using the DDPG algorithm before the aircraft reached the boundary of the sector to avoid collisions. Alternatively, Proximal Policy Optimization (PPO) methods can be used in aircraft collision avoidance, and have shown a certain level of performance [26].…”

“…The use of a discrete state space would cause a waste of airspace resources, and the whole raster area may become a no-fly zone due to some small buildings, thus reducing the space available for UAV flights. (2) The dimensional advantage is also an important factor in measuring the performance of the method, with most existing UAV collision avoidance methods borrowing from ground traffic, and thus the dimensional range is limited to 2D, which does not match the actual operation situation in the airspace, while also limiting the UAV avoidance actions that can be selected [18,20,26]. (3) When the DRL theory is applied to mUAV collision avoidance, in the existing literature, only one UAV is regarded as the agent, and the other UAVs are regarded as dynamic obstacles without resolution ability [20]; their tracks are previously planned.…”

Sensorless and Coordination-Free Lane Switching on a Drone Road Segment—A Simulation Study

“…Ragi and Chong [25] present a partially observable Markov decision process (POMDPs) [26] approach to allow drones to track the positions of others and a target, and avoid collisions with each other. Hu et al [27] employ reinforcement learning to guide drones while avoiding obstacles. Yu et al [28] consider target tracking in an urban environment and propose a cooperative path-planning algorithm integrating both aerial and ground-based autonomous vehicles.…”

Integrated entry guidance with no-fly zone constraint using reinforcement learning and predictor-corrector technique

“…15 From existing research, deep reinforcement learning is utilized to solve powered descent and landing control of Mars or the Moon. [16][17][18] Stateof-the-art RL algorithms, Proximal Policy Optimization (PPO) and Twin Delayed Deep Deterministic Policy Gradient (TD3PG), are applied to obstacles avoidance of unmanned aerial vehicles (UAVs) 19 and missile, 20 respectively. Moreover, neural network controller trained by RL augments the robustness of existing control methods, so as to deal with complex constraints and environmental uncertainties in spacecraft rendezvous guidance, 21 proximity operations, 22 and onboard applications for lowthrust spacecraft.…”