Swarm Cooperative Navigation Using Centralized Training and Decentralized Execution

Abstract

The demand for autonomous UAV swarm operations has been on the rise following the success of UAVs in various challenging tasks. Yet conventional swarm control approaches are inadequate for coping with swarm scalability, computational requirements, and real-time performance. In this paper, we demonstrate the capability of emerging multi-agent reinforcement learning (MARL) approaches to successfully and efficiently make sequential decisions during UAV swarm collaborative tasks. We propose a scalable, real-time, MARL approach for UAV collaborative navigation where members of the swarm have to arrive at target locations at the same time. Centralized training and decentralized execution (CTDE) are used to achieve this, where a combination of negative and positive reinforcement is employed in the reward function. Curriculum learning is used to facilitate the sought performance, especially due to the high complexity of the problem which requires extensive exploration. A UAV model that highly resembles the respective physical platform is used for training the proposed framework to make training and testing realistic. The scalability of the platform to various swarm sizes, speeds, goal positions, environment dimensions, and UAV masses has been showcased in (1) a load drop-off scenario, and (2) UAV swarm formation without requiring any re-training or fine-tuning of the agents. The obtained simulation results have proven the effectiveness and generalizability of our proposed MARL framework for cooperative UAV navigation.

Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph Attention Network for UAV Swarms

“…Qie et al [33] formulated the Multi-UAV Target Assignment and Path Planning (MUTAPP) problem as a multi-agent system, and adopted the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm to solve the MUTAPP problem. For large-scale multi-UAV systems, Azzam et al [34] developed MARL-based cooperative navigation of UAV swarms via centralized training and decentralized execution (CTDE), and the method was extended to work with a large number of agents , without retraining or changing the number of agents during training. Wang et al [35] proposed a novel MARL paradigm, weighted mean field reinforcement learning, and conducted experiments on a large-scale UAV swarm confrontation environment to verify the effectiveness and scalability of the method.…”

A Comprehensive Survey on Multi-Agent Reinforcement Learning for Connected and Automated Vehicles

“…In the lazy agent problem, the agents in a team are not performing equally well, but they are still receiving the same collective reward. To address this issue, researchers have suggested various learning and non-learning methods that assign credit to each agent based on their individual contributions [ 30 , 31 , 32 , 33 , 34 , 35 ]. Interestingly, the centralized training and decentralized execution indigenously have no issues or occurrences of lazy agents.…”