Cooperative Planning for an Unmanned Combat Aerial Vehicle Fleet Using Reinforcement Learning

Abstract

In this study, reinforcement learning (RL)-based centralized path planning is performed for an unmanned combat aerial vehicle (UCAV) fleet in a human-made hostile environment. The proposed method provides a novel approach in which closing speed and approximate time-to-go terms are used in the reward function to obtain cooperative motion while ensuring no-fly-zones (NFZs) and time-of-arrival constraints. Proximal policy optimization (PPO) algorithm is used in the training phase of the RL agent. System performance is evaluated in two different cases. In case 1, the warfare environment contains only the target area, and simultaneous arrival is desired to obtain the saturated attack effect. In case 2, the warfare environment contains NFZs in addition to the target area and the standard saturated attack and collision avoidance requirements. Particle swarm optimization (PSO)-based cooperative path planning algorithm is implemented as the baseline method, and it is compared with the proposed algorithm in terms of execution time and developed performance metrics. Monte Carlo simulation studies are performed to evaluate the system performance. According to the simulation results, the proposed system is able to generate feasible flight paths in real-time while considering the physical and operational constraints such as acceleration limits, NFZ restrictions, simultaneous arrival, and collision avoidance requirements. In that respect, the approach provides a novel and computationally efficient method for solving the large-scale cooperative path planning for UCAV fleets.

UAV maneuver decision-making via deep reinforcement learning for short-range air combat

“…The usage of the PPO algorithm in short-range air combat The second improvement point is to differentiate the neural network inputs. For the basic PPO algorithm, the critic network uses the same input as the actor network, which is named state space [20] . However, the actor and critic networks play different roles in the algorithm.…”

“…The actor network gets action according to the relationship in the view of the red, but the critic network gets the evaluation value based on the state of the air combat environment, which can include the information that cannot be observed. Thus, the input for the critic network 𝑠 𝑐𝑟𝑖𝑡𝑖𝑐 is defined as [17] 𝑠 𝑐𝑟𝑖𝑡𝑖𝑐 = [𝐷, 𝜑 𝑟 , 𝜑 𝑏 , 𝑧 𝑏 − 𝑧 𝑟 , 𝜓 𝐷 , 𝜓 𝑏𝑟 , 𝜃 𝑏𝑟 , 𝑣 𝑟 − 𝑣 𝑏 , 𝐵 𝑟 , 𝐵 𝑟𝑏 ] 𝑇 , (20) where subscript 𝑟𝑏 means the red' s value minus the blue' s and 𝑏𝑟 is on the contrary, and 𝐵 𝑟 is the red' s residual blood. The critic network' s state variables are also normalized and the threshold vector 𝛿 𝑐𝑟𝑖𝑡𝑖𝑐 is selected as…”

Safe Motion Planning and Learning for Unmanned Aerial Systems

“…Accordingly, optimization and learning techniques are employed in their applications to increase the number of UAV types and extend their operation range. In [1], centralized path planning based on reinforcement learning is implemented in a combat aerial vehicle fleet in order to avoid enemy defense systems. To localize radio frequency emitting targets in the operation area, multiple UAVs were deployed in [2] using Particle Filter and Extended Kalman Filter algorithms and vision-based detection.…”

Real-Time On-the-Fly Motion Planning for Urban Air Mobility via Updating Tree Data of Sampling-Based Algorithms Using Neural Network Inference

“…We found that entropy shrinks to a small value prematurely during the training process, resulting in insufficient exploration in policy learning, therefore obtaining a defective policy. In the face of this problem, we then introduce entropy regularization [31] to Equation (3).…”