Multi-Agent Incentive Communication via Decentralized Teammate Modeling

Abstract

Effective communication can improve coordination in cooperative multi-agent reinforcement learning (MARL). One popular communication scheme is exchanging agents' local observations or latent embeddings and using them to augment individual local policy input. Such a communication paradigm can reduce uncertainty for local decision-making and induce implicit coordination. However, it enlarges agents' local policy spaces and increases learning complexity, leading to poor coordination in complex settings. To handle this limitation, this paper proposes a novel framework named Multi-Agent Incentive Communication (MAIC) that allows each agent to learn to generate incentive messages and bias other agents' value functions directly, resulting in effective explicit coordination. Our method firstly learns targeted teammate models, with which each agent can anticipate the teammate's action selection and generate tailored messages to specific agents. We further introduce a novel regularization to leverage interaction sparsity and improve communication efficiency. MAIC is agnostic to specific MARL algorithms and can be flexibly integrated with different value function factorization methods. Empirical results demonstrate that our method significantly outperforms baselines and achieves excellent performance on multiple cooperative MARL tasks.

T2MAC: Targeted and Trusted Multi-Agent Communication through Selective Engagement and Evidence-Driven Integration

“…We subjected T2MAC to rigorous testing across various MARL environments, such as Hallway, MPE, and SMAC. Compared to prominent multi-agent communication strategies like TarMac (Das et al 2019), MAIC (Yuan et al 2022), SMS (Xue et al 2022), and MASIA (Guan et al 2022), T2MAC consistently excelled in both performance and efficiency. Additionally, its versatility shone through across diverse scenarios.…”

“…Building on this foundation, both VBC (Zhang, Zhang, and Lin 2019) and TMC (Zhang, Zhang, and Lin 2020) further optimize message learning processes. Furthermore, NDQ (Wang et al 2020b) and MAIC (Yuan et al 2022) are designed to craft messages tailored for individual agents.…”

“…This approach ensures stable learning and encourages harmonized actions among agents. However, historical methods have focused on the content and timing of communication (Sukhbaatar, Szlam, and Fergus 2016;Singh, Jain, and Sukhbaatar 2018;Kim et al 2019;Wang et al 2020b;Zhang, Zhang, and Lin 2020;Yuan et al 2022). Once an agent elects to share its message, it is broadcast to the entire agent group.…”

“…Assimilating these messages aptly can enrich agents' perception of an uncertain environment, leading to more refined policies. Regrettably, the existing techniques-whether they're steeped in basic aggregation (Jiang and Lu 2018) or are more avant-garde with representation learning (Das et al 2019;Guan et al 2022)-tend to treat the fusion of information as a black box, presupposing that the policy networks can innately sift out vital data and diminish decision-making uncertainty. In this context, the information integration process might prove to be both uncertain and inefficient, especially in intricate scenarios.…”

Efficient Communication via Self-supervised Information Aggregation for Online and Offline Multi-agent Reinforcement Learning

“…Communication plays a promising role in multi-agent coordination under partial observability [13,84]. Extensive research have been made on learning communication protocols to improve performance on cooperative tasks [20,15,34,73,12,37,77,71,21]. Previous works can be divided into two categories.…”