Enhancing Automated Maneuvering Decisions in UCAV Air Combat Games Using Homotopy-Based Reinforcement Learning

Abstract

In the field of real-time autonomous decision-making for Unmanned Combat Aerial Vehicles (UCAVs), reinforcement learning is widely used to enhance their decision-making capabilities in high-dimensional spaces. These enhanced capabilities allow UCAVs to better respond to the maneuvers of various opponents, with the win rate often serving as the primary optimization metric. However, relying solely on the terminal outcome of victory or defeat as the optimization target, but without incorporating additional rewards throughout the process, poses significant challenges for reinforcement learning due to the sparse reward structure inherent in these scenarios. While algorithms enhanced with densely distributed artificial rewards show potential, they risk deviating from the primary objectives. To address these challenges, we introduce a novel approach: the homotopy-based soft actor–critic (HSAC) method. This technique gradually transitions from auxiliary tasks enriched with artificial rewards to the main task characterized by sparse rewards through homotopic paths. We demonstrate the consistent convergence of the HSAC method and its effectiveness through deployment in two distinct scenarios within a 3D air combat game simulation: attacking horizontally flying UCAVs and a combat scenario involving two UCAVs. Our experimental results reveal that HSAC significantly outperforms traditional algorithms, which rely solely on using sparse rewards or those supplemented with artificially aided rewards.