Learning Individually Inferred Communication for Multi-Agent Cooperation

Ding, Ziluo; Huang, Tiejun; Lu, Zongqing

doi:10.48550/arxiv.2006.06455

Cited by 4 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A family of MARL frameworks exploit communication [15,[40][41][42] sharing mechanism allows an agent to concurrently receive other agents' base actions and the gradient backflows at the current step, which brings more timeliness to help agents make better decisions.…”

Section: Related Workmentioning

confidence: 99%

DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning

Yang¹,

Ding²,

Jin

et al. 2021

Preprint

View full text Add to dashboard Cite

In recent years, multi-agent reinforcement learning (MARL) has presented impressive performance in various applications. However, physical limitations, budget restrictions, and many other factors usually impose constraints on a multi-agent system (MAS), which cannot be handled by traditional MARL frameworks. Specifically, this paper focuses on constrained MASes where agents work cooperatively to maximize the expected team-average return under various constraints on expected team-average costs, and develops a constrained cooperative MARL framework, named DeCOM, for such MASes. In particular, DeCOM decomposes the policy of each agent into two modules, which empowers information sharing among agents to achieve better cooperation. In addition, with such modularization, the training algorithm of DeCOM separates the original constrained optimization into an unconstrained optimization on reward and a constraints satisfaction problem on costs. DeCOM then iteratively solves these problems in a computationally efficient manner, which makes DeCOM highly scalable. We also provide theoretical guarantees on the convergence of DeCOM's policy update algorithm. Finally, we validate the effectiveness of DeCOM with various types of costs in both toy and large-scale (with 500 agents) environments.

show abstract

Section: Related Workmentioning

confidence: 99%

DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning

Yang¹,

Ding²,

Jin

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…[20] uses a bidirectional recurrent neural network for communication between multiple agents. [21] proposes a selectable point-to-point communication method, which is used to determine whether agents communicate with each other via constructing a belief vector. This kind of multi-agent algorithms needs to make a communication operation when agents are executed.…”

Section: Related Workmentioning

confidence: 99%

BGC: Multi-Agent Group Belief with Graph Clustering

Zhou¹,

Zhang²,

Tang³

et al. 2020

Preprint

View full text Add to dashboard Cite

In this paper, we introduce the group concept into multi-agent reinforcement learning. In this method, agents are divided into several groups and each group completes a specific subtask so that agents can cooperate to complete the main task. Existing methods use the communication vector to exchange information between agents. This may encounter communication redundancy. To solve this problem, we propose a MARL method based on graph clustering. It allows agents to adaptively learn group features and replaces the communication operation. In our method, agent features are divide into two types, including in-group features and individual features. They represent the generality and differences between agents, respectively. Based on the graph attention network(GAT), we introduce the graph clustering method as a punishment to optimize agent group feature. Then these features are used to generate individual Q value. To overcome the consistent problem brought by GAT, we introduce the split loss to distinguish agent features. Our method is easy to convert into the CTDE framework via using Kullback-Leibler divergence method. Empirical results are evaluated on a challenging set of StarCraft II micromanagement tasks. The result shows that our method outperforms existing multi-agent reinforcement learning methods and the performance increases with the number of agents increasing.

show abstract

Offline Decentralized Multi-Agent Reinforcement Learning

Jiang

2021

Preprint

View full text Add to dashboard Cite

In many real-world multi-agent cooperative tasks, due to high cost and risk, agents cannot interact with the environment and collect experiences during learning, but have to learn from offline datasets. However, the transition probabilities calculated from the dataset can be much different from the transition probabilities induced by the learned policies of other agents, creating large errors in value estimates. Moreover, the experience distributions of agents' datasets may vary wildly due to diverse behavior policies, causing large difference in value estimates between agents. Consequently, agents will learn uncoordinated suboptimal policies. In this paper, we propose MABCQ, which exploits value deviation and transition normalization to modify the transition probabilities. Value deviation optimistically increases the transition probabilities of high-value next states, and transition normalization normalizes the biased transition probabilities of next states. They together encourage agents to discover potential optimal and coordinated policies. Mathematically, we prove the convergence of Q-learning under the non-stationary transition probabilities after modification. Empirically, we show that MABCQ greatly outperforms baselines and reduces the difference in value estimates between agents.Preprint. Under review.

show abstract

Learning Individually Inferred Communication for Multi-Agent Cooperation

Cited by 4 publications

References 9 publications

DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning

DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning

BGC: Multi-Agent Group Belief with Graph Clustering

Offline Decentralized Multi-Agent Reinforcement Learning

Contact Info

Product

Resources

About