Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph Attention Network for UAV Swarms

Abstract

Multiple unmanned aerial vehicles (Multi-UAV) systems have recently demonstrated significant advantages in some real-world scenarios, but the limited communication range of UAVs poses great challenges to multi-UAV collaborative decision-making. By constructing the multi-UAV cooperation problem as a multi-agent system (MAS), the cooperative decision-making among UAVs can be realized by using multi-agent reinforcement learning (MARL). Following this paradigm, this work focuses on developing partially observable MARL models that capture important information from local observations in order to select effective actions. Previous related studies employ either probability distributions or weighted mean field to update the average actions of neighborhood agents. However, they do not fully consider the feature information of surrounding neighbors, resulting in a local optimum often. In this paper, we propose a novel partially multi-agent reinforcement learning algorithm to remedy this flaw, which is based on graph attention network and partially observable mean field and is named as the GPMF algorithm for short. GPMF uses a graph attention module and a mean field module to describe how an agent is influenced by the actions of other agents at each time step. The graph attention module consists of a graph attention encoder and a differentiable attention mechanism, outputting a dynamic graph to represent the effectiveness of neighborhood agents against central agents. The mean field module approximates the effect of a neighborhood agent on a central agent as the average effect of effective neighborhood agents. Aiming at the typical task scenario of large-scale multi-UAV cooperative roundup, the proposed algorithm is evaluated based on the MAgent framework. Experimental results show that GPMF outperforms baselines including state-of-the-art partially observable mean field reinforcement learning algorithms, providing technical support for large-scale multi-UAV coordination and confrontation tasks in communication-constrained environments.

Dynamic traffic signal control using mean field multi‐agent reinforcement learning in large scale road‐networks

“…In previous works, the proof of mean field Q‐learning converges to the Nash Q ‐values has been shown in ref. [22], and the convergence of MFQ with double estimators has been proved by ref. [24].…”

“…Based on this, the mean field approximation is proposed in ref. [22] to reduce the action space's dimension and to handle the convergence problem in MARL. In this theory, the action is encoded by one‐hot coding for computing convenience.…”

“…In ref. [22], the mean field theory is introduced into the reinforcement learning to effectively decompose the dimension of joint action. The authors proposed a mean field Q‐learning (MFQ) to firstly solve the Ising model via model‐free reinforcement learning methods, where the experiment includes thousands of agents.…”

Comprehensive Review of Drones Collision Avoidance Schemes: Challenges and Open Issues

“…As a result, traditional MARL experiences scaling problems, particularly with exponential agent increments. To address this constraint, current research has focused on addressing scaling issues in MARL by adding mean field theory [172].…”

Multi-agent reinforcement learning for complex sequential decision-making

“…The objective of each agent is to maximize its total return R i = ∞ t=0 γ t r t i . In Markov Games settings, agents can cooperate [25,71,72], compete with other agents for resources or rewards [18,73] or even betray the cooperation [74]. Many MARL methods solve Markov Games problems [25,71,[75][76][77][78][79][80][81], among which MADDPG [25], MF-Q [71] and PSRO [75] are representative methods.…”