Counterfactual Multi-Agent Policy Gradients

Abstract

Many real-world problems, such as network packet routing and the coordination of autonomous vehicles, are naturally modelled as cooperative multi-agent systems. There is a great need for new reinforcement learning methods that can efficiently learn decentralised policies for such systems. To this end, we propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients. COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies. In addition, to address the challenges of multi-agent credit assignment, it uses a counterfactual baseline that marginalises out a single agent's action, while keeping the other agents' actions fixed. COMA also uses a critic representation that allows the counterfactual baseline to be computed efficiently in a single forward pass. We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability. COMA significantly improves average performance over other multi-agent actorcritic methods in this setting, and the best performing agents are competitive with state-of-the-art centralised controllers that get access to the full state.

Survey of recent multi-agent reinforcement learning algorithms utilizing centralized training

“…(iii) Counterfactual Multi-Agent Policy Gradients (COMA) is a CLDE based method which uses a single centralized critic for all the agents to estimate the Q-function and decentralised actors to optimise the agents' policies. 42 It addresses the credit assignment problem in multi-agent systems which earlier has been studied using difference rewards, an approach shaping the global reward such that agents are rewarded or penalized based on their contributions to the system's performance. These rewards are often found by estimating a reward function or using simulation, approaches which are not always trivial.…”

Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition

“…A branch of methods investigates factorizable Q functions (Sunehag et al, 2017;Rashid et al, 2018;Mahajan et al, 2019;Son et al, 2019) where the team Q is decomposed into individual utility functions. Some other methods adopt the actor-critic method where only the critic is centralized (Foerster et al, 2017;Lowe et al, 2017). However, most CTDE methods by structure require fixedsize teams and are often applied to homogeneous teams.…”

Joint Resource Management for MC-NOMA: A Deep Reinforcement Learning Approach

“…In contrast, a total optimization objective value cannot be directly applied as the joint reward r p,jo m,t of each PA-agent for two reasons: 1) A global reward makes it difficult for each agent to deduce its individual contribution. The gradient computed for each actor does not explicitly reason about how the agent's actions contribute to the global reward [35]; and 2) different users (i.e., different agents) can have different weights to account for different priorities. The final reward r p m,t of PAagent m is r p m,t = r p,int m,t +r p,jo m,t .…”