TarMAC: Targeted Multi-Agent Communication

Das, Abhishek; Theophile, Gervet,; Romoff, Joshua; Batra, Dhruv; Parikh, Devi; Rabbat, Michael; Pineau, Joëlle

doi:10.48550/arxiv.1810.11187

Cited by 12 publications

(17 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their training algorithm is an extension of the deep recurrent Q-learning (DRQN) (Hausknecht and Stone, 2015), which combines RNN and deep Q-learning (Mnih et al, 2015). Following Foerster et al (2016), various works (Jorge et al, 2016;Sukhbaatar et al, 2016;Havrylov and Titov, 2017;Das et al, 2017;Peng et al, 2017;Mordatch and Abbeel, 2018;Celikyilmaz et al, 2018;Das et al, 2018;Lazaridou et al, 2018;Cogswell et al, 2019) have proposed a variety of neural network architectures to foster communication among agents. These works combine single-agent RL methods with novel developments in deep learning, and demonstrate their performances via empirical studies.…”

Section: Learning To Communicatementioning

confidence: 99%

Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms

Zhang¹,

Yang²,

Başar³

2019

Preprint

154

View full text Add to dashboard Cite

Recent years have witnessed significant advances in reinforcement learning (RL), which has registered great success in solving various sequential decision-making problems in machine learning. Most of the successful RL applications, e.g., the games of Go and Poker, robotics, and autonomous driving, involve the participation of more than one single agent, which naturally fall into the realm of multi-agent RL (MARL), a domain with a relatively long history, and has recently re-emerged due to advances in single-agent RL techniques. Though empirically successful, theoretical foundations for MARL are relatively lacking in the literature. In this chapter, we provide a selective overview of MARL, with focus on algorithms backed by theoretical analysis. More specifically, we review the theoretical results of MARL algorithms mainly within two representative frameworks, Markov/stochastic games and extensive-form games, in accordance with the types of tasks they address, i.e., fully cooperative, fully competitive, and a mix of the two. We also introduce several significant but challenging applications of these algorithms. Orthogonal to the existing reviews on MARL, we highlight several new angles and taxonomies of MARL theory, including learning in extensive-form games, decentralized MARL with networked agents, MARL in the mean-field regime, (non-)convergence of policy-based methods for learning in games, etc. Some of the new angles extrapolate from our own research endeavors and interests. Our overall goal with this chapter is, beyond providing an assessment of the current state of the field on the mark, to identify fruitful future research directions on theoretical studies of MARL. We expect this chapter to serve as continuing stimulus for researchers interested in working on this exciting while challenging topic.Recent years have witnessed sensational advances of reinforcement learning (RL) in many prominent sequential decision-making problems, such as playing the game of Go (Silver

show abstract

Section: Learning To Communicatementioning

confidence: 99%

Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms

Zhang¹,

Yang²,

Başar³

2019

Preprint

154

View full text Add to dashboard Cite

show abstract

“…Attention in this work encodes agents' individual observation before passing centralized communication channel. With centralized value estimate, TarMAC [9] is a targeted communication architecture to generate agents' internal state representations as input of centralized critic. Iqbal & Sha [25] introduced an attention-based critic to select agents in centralized training.…”

Section: Related Workmentioning

confidence: 99%

“…Considering the scale of an agent team in the real scenarios, centralized training is challenging considering training stability and high computation complexity with large computational costs. Jiang & Lu [8] and Das et al [9] both proposed attention based communication protocol to exchange messages in MARL domains [8], [9], but these approaches did not consider agents' behavior knowledge. Communication methods in MARL domains are always designed for solve the problem of efficient sharing partial observation information.…”

Section: Introductionmentioning

confidence: 99%

Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning

Liang,

2020

Preprint

View full text Add to dashboard Cite

Multi-agent reinforcement learning is a standard framework for modeling multi-agent interactions applied in realworld scenarios. Inspired by experience sharing in human groups, learning knowledge parallel reusing between agents can potentially promote team learning performance, especially in multitask environments. When all agents interact with the environment and learn simultaneously, how each independent agent selectively learns from other agents' behavior knowledge is a problem that we need to solve. This paper proposes a novel knowledge transfer framework in MARL, PAT (Parallel Attentional Transfer). We design two acting modes in PAT, student mode and self-learning mode. Each agent in our approach trains a decentralized student actor-critic to determine its acting mode at each time step. When agents are unfamiliar with the environment, the shared attention mechanism in student mode effectively selects learning knowledge from other agents to decide agents' actions. PAT outperforms state-of-the-art empirical evaluation results against the prior advising approaches. Our approach not only significantly improves team learning rate and global performance, but also is flexible and transferable to be applied in various multi-agent systems.

show abstract

“…Multi-Actor-Attention-Critic (MAAC) is proposed in [19] to aggregate information using attention mechanism from all the other agents. Similarly, [11], [13], [20] also employ the attention mechanism to learn a representation for the action-value function. However, the communication graphs used there are either dense or ad-hoc (k nearest neighbors), which makes the learning difficult.…”

Section: A Related Workmentioning

confidence: 99%

Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph

Shen

How

2020

2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

The complexity of multiagent reinforcement learning (MARL) in multiagent systems increases exponentially with respect to the agent number. This scalability issue prevents MARL from being applied in large-scale multiagent systems. However, one critical feature in MARL that is often neglected is that the interactions between agents are quite sparse. Without exploiting this sparsity structure, existing works aggregate information from all of the agents and thus have a high sample complexity. To address this issue, we propose an adaptive sparse attention mechanism by generalizing a sparsity-inducing activation function. Then a sparse communication graph in MARL is learned by graph neural networks based on this new attention mechanism. Through this sparsity structure, the agents can communicate in an effective as well as efficient way via only selectively attending to agents that matter the most and thus the scale of the MARL problem is reduced with little optimality compromised. Comparative results show that our algorithm can learn an interpretable sparse structure and outperforms previous works by a significant margin on applications involving a large-scale multiagent system.

show abstract

TarMAC: Targeted Multi-Agent Communication

Cited by 12 publications

References 0 publications

Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms

Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms

Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning

Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph

Contact Info

Product

Resources

About