A Multi-UCAV Cooperative Decision-Making Method Based on an MAPPO Algorithm for Beyond-Visual-Range Air Combat

Liu, Xiaoxiong; Yin, Yilong; Su, Yuzhan; Ming, Ruichen

doi:10.3390/aerospace9100563

Cited by 19 publications

(6 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Researchers have been exploring as well the potential of combining DRL with UCAVs' air combat decision-making [18,20,25]. For instance, Li [20] proposed a model for UCAV autonomous manoeuvre decision-making in short-range air combat, based on the multi-step double deep Q-network (MS-DDQN) algorithm.…”

Section: Applications To Unmanned Combat Aerial Vehicles (Ucavs)mentioning

confidence: 99%

“…To address the limitations of traditional methods such as poor flexibility and weak decisionmaking ability, some researchers have proposed the use of deep learning for manoeuvring [18]. Liu et al [25] proposed a multi-UCAV cooperative decision-making method based on a multi-agent proximal policy optimization (MAPPO) algorithm. DRL has been used by Hu et al [26] to plan beyond-visual-range air combat tactics and by Wang et al [27] to quantify the relationship between the flight agility of a UCAV and its shortrange aerial combat effectiveness.…”

Section: Applications To Unmanned Combat Aerial Vehicles (Ucavs)mentioning

confidence: 99%

“…This is the main motivation for the present research. A related motivation lies in the possibility of implementing more effective AI-assisted pilot training procedures for highprecision tasks, such as dogfights and formation flights [20,[25][26][27].…”

Section: Deep Reinforcement Learning Approach Applied To Flight Controlmentioning

confidence: 99%

See 2 more Smart Citations

A deep reinforcement learning control approach for high-performance aircraft

De Marco,

D’Onza,

Manfredi

2023

Nonlinear Dyn

View full text Add to dashboard Cite

This research introduces a flight controller for a high-performance aircraft, able to follow randomly generated sequences of waypoints, at varying altitudes, in various types of scenarios. The study assumes a publicly available six-degree-of-freedom (6-DoF) rigid aeroplane flight dynamics model of a military fighter jet. Consolidated results in artificial intelligence and deep reinforcement learning (DRL) research are used to demonstrate the capability to make certain manoeuvres AI-based fully automatic for a high-fidelity nonlinear model of a fixed-wing aircraft. This work investigates the use of a deep deterministic policy gradient (DDPG) controller agent, based on the successful applications of the same approach to other domains. In the particular application to flight control presented here, the effort has been focused on the design of a suitable reward function used to train the agent to achieve some given navigation tasks. The trained controller is successful on highly coupled manoeuvres, including rapid sequences of turns, at both low and high flight Mach numbers, in simulations reproducing a prey–chaser dogfight scenario. Robustness to sensor noise, atmospheric disturbances, different initial flight conditions and varying reference signal shapes is also demonstrated.

show abstract

Section: Applications To Unmanned Combat Aerial Vehicles (Ucavs)mentioning

confidence: 99%

Section: Applications To Unmanned Combat Aerial Vehicles (Ucavs)mentioning

confidence: 99%

See 1 more Smart Citation

A deep reinforcement learning control approach for high-performance aircraft

De Marco,

D’Onza,

Manfredi

2023

Nonlinear Dyn

View full text Add to dashboard Cite

show abstract

“…Li et al proposed a meta twin delayed deep deterministic policy gradient (Meta-TD3) to realize the control of UAV maneuvering for target tracking and enable a UAV to quickly adapt to an uncertain environment [23]. Furthermore, in order to improve the cooperative capabilities of the UCAV swarm, multi-agent deep reinforcement learning (MADRL) algorithms were applied to large-scale air combat [24,25]. Based on centralized training with decentralized execution, MADRL provides a UCAV swarm with a high level of robustness.…”

Section: Related Workmentioning

confidence: 99%

Memory-Enhanced Twin Delayed Deep Deterministic Policy Gradient (ME-TD3)-Based Unmanned Combat Aerial Vehicle Trajectory Planning for Avoiding Radar Detection Threats in Dynamic and Unknown Environments

Li,

Zhang,

Liu

2023

Remote Sensing

View full text Add to dashboard Cite

Unmanned combat aerial vehicle (UCAV) trajectory planning to avoid radar detection threats is a complicated optimization problem that has been widely studied. The rapid changes in Radar Cross Sections (RCSs), the unknown cruise trajectory of airborne radar, and the uncertain distribution of radars exacerbate the complexity of this problem. In this paper, we propose a novel UCAV trajectory planning method based on deep reinforcement learning (DRL) technology to overcome the adverse impacts caused by the dynamics and randomness of environments. A predictive control model is constructed to describe the dynamic characteristics of the UCAV trajectory planning problem in detail. To improve the UCAV’s predictive ability, we propose a memory-enhanced twin delayed deep deterministic policy gradient (ME-TD3) algorithm that uses an attention mechanism to effectively extract environmental patterns from historical information. The simulation results show that the proposed method can successfully train UCAVs to carry out trajectory planning tasks in dynamic and unknown environments. Furthermore, the ME-TD3 algorithm outperforms other classical DRL algorithms in UCAV trajectory planning, exhibiting superior performance and adaptability.

show abstract

“…Li developed a MARL-based algorithm with a centralized critic, an actor using the gate recurrent unit model, and a dual experience playback mechanism for solving multi-UAV collaborative decision-making problems and evaluated it through simulations in a multi-UAV air combat environment [23]. In addition, Liu designed a missile attack zone model and combined it with the multi-agent proximal policy optimization (MAPPO) method to achieve beyond-visual-range air combat among multiple unmanned aerial vehicles [24]. The references [23,24] have investigated the MCAC decision-making problem concerning discrete action spaces.…”

Section: Introductionmentioning

confidence: 99%

Multi-UAV Cooperative Air Combat Decision-Making Based on Multi-Agent Double-Soft Actor-Critic

Wang

Zhou

et al. 2023

Aerospace

View full text Add to dashboard Cite

Multiple unmanned aerial vehicle (multi-UAV) cooperative air combat, which is an important form of future air combat, has high requirements for the autonomy and cooperation of unmanned aerial vehicles. Therefore, it is of great significance to study the decision-making method of multi-UAV cooperative air combat since the conventional methods are challenging to solve the high complexity and highly dynamic cooperative air combat problems. This paper proposes a multi-agent double-soft actor-critic (MADSAC) algorithm for solving the cooperative decision-making problem of multi-UAV. The MADSAC achieves multi-UAV cooperative air combat by treating the problem as a fully cooperative game using a decentralized partially observable Markov decision process and a centrally trained distributed execution framework. The use of maximum entropy theory in the update process makes the method more exploratory. Meanwhile, MADSAC uses double-centralized critics, target networks, and delayed policy updates to solve the overestimation and error accumulation problems effectively. In addition, the double-centralized critics based on the attention mechanism improve the scalability and learning efficiency of MADSAC. Finally, multi-UAV cooperative air combat experiments validate the effectiveness of MADSAC.

show abstract

A Multi-UCAV Cooperative Decision-Making Method Based on an MAPPO Algorithm for Beyond-Visual-Range Air Combat

Cited by 19 publications

References 29 publications

A deep reinforcement learning control approach for high-performance aircraft

A deep reinforcement learning control approach for high-performance aircraft

Memory-Enhanced Twin Delayed Deep Deterministic Policy Gradient (ME-TD3)-Based Unmanned Combat Aerial Vehicle Trajectory Planning for Avoiding Radar Detection Threats in Dynamic and Unknown Environments

Multi-UAV Cooperative Air Combat Decision-Making Based on Multi-Agent Double-Soft Actor-Critic

Contact Info

Product

Resources

About