Learning Nearly Decomposable Value Functions Via Communication Minimization

Abstract

Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multi-agent systems. However, existing methods have been focusing on learning fully decentralized value function, which are not efficient for tasks requiring communication. To address this limitation, this paper presents a novel framework for learning nearly decomposable value functions with communication, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination. This framework hybridizes value function factorization learning and communication learning by introducing two information-theoretic regularizers. These regularizers are maximizing mutual information between decentralized Q functions and communication messages while minimizing the entropy of messages between agents. We show how to optimize these regularizers in a way that is easily integrated with existing value function factorization methods such as QMIX. Finally, we demonstrate that, on the StarCraft unit micromanagement benchmark, our framework significantly outperforms baseline methods and allows to cut off more than 80% communication without sacrificing the performance. The video of our experiments is available at https://sites.google.com/view/ndvf.

QPLEX: Duplex Dueling Multi-Agent Q-Learning

“…where K is the number of attention heads, λ i,k (τ , a) and φ i,k (τ ) are attention weights activated by a sigmoid regularizer, and υ k (τ ) > 0 is a positive key of each head. This sigmoid activation of λ i brings sparsity to the credit assignment of the joint advantage function to individuals, which enables efficient multi-agent learning [19].…”

Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients

“…The closest paper to our work is NDQ [44], which also utilizes latent variables to represent the information but as the communication messages during the decentralized agents execution. Although we both consider the information extraction as an information bottleneck problem, there are several key differences between our work and NDQ: (I) NDQ is a value-based method, while our work is a policy-based method under the soft-actor-critic framework.…”

“…The main shortcoming is that the computation of the mutual information is computationally challenging. Inspired by the recent advancement in Bayesian inference and variational auto-encoder [17,28,44], we propose a novel way of representing it by utilizing latent vectors from variational inference models using information theoretical regularization method, and then derive the evidence lower bound (ELBO) of its objective.…”

“…It is shown to significantly improve the power of value function factorization. Since we utilize such latent state information sharing only in centralized critic, the CTDE assumptions are preserved without affecting fully decentralized decision making, unlike previous work introducing global communications [44]. Further, we note that combining actor-critic framework with value decomposition in LSF-SAC offers a way to decouple the decision making of individual agents (through separate policy networks) from value function networks, while also allowing the maximization of entropy to enhance its stability and exploration.…”

LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates

“…ROMA [30] constructs a stochastic role embedding space to lead agents to different policies based on different roles. NDQ [32] learns a message representation to achieve expressive and succinct communication. RODE [31] uses an action encoder to learn action representations and applies clustering methods to decompose joint action spaces into restricted role action spaces to reduce the policy search space.…”