Countering a Drone in a 3D Space: Analyzing Deep Reinforcement Learning Methods

Abstract

Unmanned aerial vehicles (UAV), also known as drones have been used for a variety of reasons and the commercial drone market growth is expected to reach remarkable levels in the near future. However, some drone users can mistakenly or intentionally fly into flight paths at major airports, flying too close to commercial aircraft or invading people’s privacy. In order to prevent these unwanted events, counter-drone technology is needed to eliminate threats from drones and hopefully they can be integrated into the skies safely. There are various counter-drone methods available in the industry. However, a counter-drone system supported by an artificial intelligence (AI) method can be an efficient way to fight against drones instead of human intervention. In this paper, a deep reinforcement learning (DRL) method has been proposed to counter a drone in a 3D space by using another drone. In a 2D space it is already shown that the deep reinforcement learning method is an effective way to counter a drone. However, countering a drone in a 3D space with another drone is a very challenging task considering the time required to train and avoid obstacles at the same time. A Deep Q-Network (DQN) algorithm with dueling network architecture and prioritized experience replay is presented to catch another drone in the environment provided by an Airsim simulator. The models have been trained and tested with different scenarios to analyze the learning progress of the drone. Experiences from previous training are also transferred before starting a new training by pre-processing the previous experiences and eliminating those considered as bad experiences. The results show that the best models are obtained with transfer learning and the drone learning progress has been increased dramatically. Additionally, an algorithm which combines imitation learning and reinforcement learning is implemented to catch the target drone. In this algorithm, called deep q-learning from demonstrations (DQfD), expert demonstrations data and self-generated data by the agent are sampled and the agent continues learning without overwriting the demonstration data. The main advantage of this algorithm is to accelerate the learning process even if there is a small amount of demonstration data.

A Comprehensive Survey of Unmanned Aerial Vehicles Detection and Classification Using Machine Learning Approach: Challenges, Solutions, and Future Directions

“…Later, a deep object detector and search area proposal algorithm are used to predict target drones. Another study in [201] proposed a deep Q-network-based method to counter drones in 3D space. The authors used EfficientNet-B0, a sub-version of EiffientNet, to detect drones that can capture small objects.…”

Explainability of Deep Reinforcement Learning Method with Drones

“…The details of the agent's neural network can be seen in detail in the following sections III-B and it is also presented in our previous research [13].…”

Deep Reinforcement Learning Tf-Agent-Based Object Tracking With Virtual Autonomous Drone in a Game Engine

“…B. Abdelkader et al [23] propose RL-based drone elevation control on a Python-Unity integrated simulation framework to achieve a stable user diagram protocol (UDP) with the suggested algorithm. E. Ç etin [24] proposes counting drones in a 3D space with several DRL methods present to count drones with another drone in the environment provided by an AirSim simulator.…”