Communication Learning via Backpropagation in Discrete Channels with Unknown Noise

Abstract

This work focuses on multi-agent reinforcement learning (RL) with inter-agent communication, in which communication is differentiable and optimized through backpropagation. Such differentiable approaches tend to converge more quickly to higher-quality policies compared to techniques that treat communication as actions in a traditional RL framework. However, modern communication networks (e.g., Wi-Fi or Bluetooth) rely on discrete communication channels, for which existing differentiable approaches that consider real-valued messages cannot be directly applied, or require biased gradient estimators. Some works have overcome this problem by treating the message space as an extension of the action space, and use standard RL to optimize message selection, but these methods tend to converge slower and to inferior policies. In this paper, we propose a stochastic message encoding/decoding procedure that makes a discrete communication channel mathematically equivalent to an analog channel with additive noise, through which gradients can be backpropagated. Additionally, we introduce an encryption step for use in noisy channels that forces channel noise to be message-independent, allowing us to compute unbiased derivative estimates even in the presence of unknown channel noise. To the best of our knowledge, this work presents the first differentiable communication learning approach that can compute unbiased derivatives through channels with unknown noise. We demonstrate the effectiveness of our approach in two example multi-robot tasks: a path finding and a collaborative search problem. There, we show that our approach achieves learning speed and performance similar to differentiable communication learning with real-valued messages (i.e., unlimited communication bandwidth), while naturally handling more realistic real-world communication constraints. Content Areas: Multi-Agent Communication, Reinforcement Learning.

A Survey of Multi-Agent Reinforcement Learning with Communication

“…Agent-Entity Graph [29] also uses distance to measure nearby agents while as long as two agents are close to each other will be allowed to communicate. LSC [45] enables agents within a cluster [38]; MAGNet-SA-GS-MG [46]; Agent-Entity Graph [29]; LSC [45]; NeurComm [32]; IP [33]; FlowComm [14]; GAXNet [44]; Other Agents DIAL [11]; RIAL [11]; CommNet [12]; BiCNet [36]; TarMAC [20]; MADDPG-M [47]; IC3Net [24]; SchedNet [25]; DCC-MD [48]; VBC [39]; Diff Discrete [49]; I2C [28]; IS [34]; ETCNet [27]; Variablelength Coding [43]; TMC [40]; Proxy MS-MARL-GCM [50]; ATOC [23]; MD-MADDPG [37]; IMAC [21]; GA-Comm [22]; Gated-ACML [26]; HAMMER [51]; MAGIC [13]; radius to decide whether to become a leader agent. Then all non-leader agents in a cluster will communicate with the only one leader agent.…”

“…DIAL [11], RIAL [11], CommNet [12], and BiCNet [36] learn a communication protocol which connect all agents together. Diff Discrete [49] and Variable-length Coding [43] consider two-agent cases while do not learn to block messages from each other. TarMAC [20] and IS [34] learn meaningful messages while using a broadcast way to share messages thus still using full communication.…”

“…Types Subtypes Methods Predefined Full Communication DIAL [11]; RIAL [11]; CommNet [12]; BiCNet [36]; MS-MARL-GCM [50]; TarMAC [20]; MD-MADDPG [37]; DCC-MD [48]; IMAC [21]; Diff Discrete [49]; IS [34]; Variable-length Coding [43]; HAMMER [51];…”

FCMNet: Full Communication Memory Net for Team-Level Cooperation in Multi-Agent Systems

“…5.2.1 Multi-Agent Pathfinding with Individual Rewards. To further illustrate the robustness of FCMNet, we consider the simple partiallyobservable, cooperative multi-agent pathfinding task introduced in [9] (Hidden-Goal Path-Finding), with a team of 𝑛 = 5 agents. In this task, each agent has a unique target location it needs to reach as soon as possible, and whose position may change randomly at every timestep.…”

“…We experimentally evaluate our proposed model on a range of unit micromanagement tasks in the StarCraft II Multi-Agent Challenge [21], as well as on a partially-observable multi-agent pathfinding task [9]. Our results show that FCMNet outperforms existing state-of-the-art CL methods in all StarCraft II micromanagement tasks and value decomposition methods in certain tasks.…”