A Vision Based Deep Reinforcement Learning Algorithm for UAV Obstacle Avoidance

Abstract

Integration of reinforcement learning with unmanned aerial vehicles (UAVs) to achieve autonomous flight has been an active research area in recent years. An important part focuses on obstacle detection and avoidance for UAVs navigating through an environment. Exploration in an unseen environment can be tackled with Deep Q-Network (DQN). However, value exploration with uniform sampling of actions may lead to redundant states, where often the environments inherently bear sparse rewards. To resolve this, we present two techniques for improving exploration for UAV obstacle avoidance. The first is a convergence-based approach that uses convergence error to iterate through unexplored actions and temporal threshold to balance exploration and exploitation. The second is a guidance-based approach using a Domain Network which uses a Gaussian mixture distribution to compare previously seen states to a predicted next state in order to select the next action. Performance and evaluation of these approaches were implemented in multiple 3-D simulation environments, with variation in complexity. The proposed approach demonstrates a two-fold improvement in average rewards compared to state of the art.

Interpretable UAV Collision Avoidance using Deep Reinforcement Learning

“…The agent increased its rewards and steps just by staying at one place and rotating indefinitely. Also, while comparing our work with [18], we noted that the training times using the Guidance Training algorithm were a lot larger and that it took days to train for a 1000 episodes. In contrast, the SAM model could be trained for 8k episodes in a few hours.…”

“…However, the use of attention based mechanisms including transformers have been used to replace LSTMs and GRUs due to their inherent limitations [27]. Moreover, [18] developed an autonomous multi-rotor flight DRL algorithm that enabled collision avoidance under novel scenarios while encouraging exploration. The algorithm used depth images as input while flying across an enclosed space with the intention of objective the of maximizing the number of steps taken before a crash.…”

“…The output of the CNN is a set of Q-values equaling the number of actions. Based on the Q value, the algorithm chooses an action that is then used to maximize the expected reward [18]. The disadvantage of using CNNs is that they are generally not invariant to deformations or rotations.…”

Towards monocular vision-based autonomous flight through deep reinforcement learning

“…Although the algorithm has good performance, constructing numerous real world-like environments (Sadeghi & Levine, 2016) requires significant efforts and time. He et al (2020); Roghair et al (2021) use real-like virtual simulators to train the obstacle avoidance algorithms but they are not evaluated in the real environments.…”

“…Especially in the computer vision field, the level of situation awareness based on visual information has been dramatically improved. For example, AlexNet (Krizhevsky et al, 2012) has significantly improved the object classification performance by intro-J o u r n a l P r e -p r o o f vironments (e.g., depth images) during the training (Loquercio et al, 2021;Ramezani Dooraki & Lee, 2018;Wu et al, 2018), or use virtual simulators that are similar to real environments (He et al, 2020;Roghair et al, 2021;Sadeghi & Levine, 2016). Ahn & Song (2020) trained the robot arm grasping policy in the simulation using a vision sensor and deploy the learned policy in the physical worlds with additional training in real-world environments.…”