Automated Enemy Avoidance of Unmanned Aerial Vehicles Based on Reinforcement Learning

Abstract

This paper focuses on one of the collision avoidance scenarios for unmanned aerial vehicles (UAVs), where the UAV needs to avoid collision with the enemy UAV during its flying path to the goal point. Such a type of problem is defined as the enemy avoidance problem in this paper. To deal with this problem, a learning based framework is proposed. Under this framework, the enemy avoidance problem is formulated as a Markov Decision Process (MDP), and the maneuver policies for the UAV are learned based on a temporal-difference reinforcement learning method called Sarsa. To handle the enemy avoidance problem in continuous state space, the Cerebellar Model Arithmetic Computer (CMAC) function approximation technique is embodied in the proposed framework. Furthermore, a hardware-in-the-loop (HITL) simulation environment is established. Simulation results show that the UAV agent can learn a satisfying policy under the proposed framework. Comparing with the random policy and the fixed-rule policy, the learned policy can achieve a far higher possibility in reaching the goal point without colliding with the enemy UAV.

Deep Reinforcement Learning With Application to Air Confrontation Intelligent Decision-Making of Manned/Unmanned Aerial Vehicle Cooperative System

“…Classical reinforcement learning is difficult or impossible to traverse all cases in the face of high dimension of state and action space, which may result in slow convergence of the algorithm or the inability to learn reasonable strategies. An effective way to solve the above problems is to use the method of function approximation to express the value function or strategy explicitly [21]. For the complex nonlinear function, the deep neural network has a better approximate effect, so it has become a trend to introduce the deep neural network as a tool into RL for approximating the value function or strategy function in recent years [20].…”

“…According to the requirement of discrete control, deep Q network (DQN), a traditional DRL method for solving discrete space was applied. Second, the efficiency of RL should still be further improved although numerous researchers have focused on the issue [21], [22]. Typically, DDPG, an algorithm for solving continuous controlling models was proposed by DeepMind in 2016 [17].…”

Traffic Light Cycle Configuration of Single Intersection Based on Modified Q-Learning

“…Reinforced learning has been used in traffic light configuration to help us find the best state by rewarding the actions chosen by the agent. Common methods of Reinforcement Learning are Q-learning [24,25], Sarsa [26,27] and Policy Gradients [28,29]. Among them, Q-learning is a prominent method in the field of control, which can reduce the risks and burdens caused by manual control.…”

Special Feature on Advanced Mobile Robotics

“…Recently, control of UAVs (unmanned aerial vehicles) has been a challenging problem and an example of a task is that a robot needs to avoid collision with an enemy UAV in its flying path to the goal. Cheng et al [19] formulated this as a Markov decision process and applied temporal-difference reinforcement learning to the robot control. The learned policy can achieve a good performance to reach the goal without colliding with the enemy.…”