Learning to Communicate Implicitly by Actions

Abstract

In situations where explicit communication is limited, human collaborators act by learning to: (i) infer meaning behind their partner's actions, and (ii) convey private information about the state to their partner implicitly through actions. The first component of this learning process has been well-studied in multi-agent systems, whereas the second — which is equally crucial for successful collaboration — has not. To mimic both components mentioned above, thereby completing the learning process, we introduce a novel algorithm: Policy Belief Learning (PBL). PBL uses a belief module to model the other agent's private information and a policy module to form a distribution over actions informed by the belief module. Furthermore, to encourage communication by actions, we propose a novel auxiliary reward which incentivizes one agent to help its partner to make correct inferences about its private information. The auxiliary reward for communication is integrated into the learning of the policy module. We evaluate our approach on a set of environments including a matrix game, particle environment and the non-competitive bidding problem from contract bridge. We show empirically that this auxiliary reward is effective and easy to generalize. These results demonstrate that our PBL algorithm can produce strong pairs of agents in collaborative games where explicit communication is disabled.

Communicating via Markov Decision Processes

“…We describe two classes of algorithms that we believe are most closely related to this GME. The first regards modeling the private information of other agents [39,1,31]. These approaches target a more general problem setting than the one we investigate here.…”

BridgeHand2Vec Bridge Hand Representation

“…to teach a non-standard bidding system, can be problematic. • Many learning algorithms in bridge bidding use (or attempt to use) partner's hand estimation [27,26,18,7,22]. BridgeHand2V ec provides better cost functions and methods for assessing estimation quality.…”

“…Currently, bridge programs use rule sets. However, experiments using reinforcement learning [25,18,11,7,22,21] indicate that it is possible to improve here by creating a custom system, but this poses problems with revealing arrangements.…”

“…The proposed approach of meaningful representation of hidden states can be generalized to other games and algorithms with hidden state estimation or belief modelling (e.g. Policy Belief Learning [22]).…”

Differentiable Multi-Agent Actor-Critic for Multi-Step Radiology Report Summarization

“…MARL often has centralized training and decentralized execution (Foerster et al, 2016;Kraemer and Banerjee, 2016). There are several protocols for MARL training, such as sharing parameters between agents and explicit (Foerster et al, 2016(Foerster et al, , 2018Sukhbaatar et al, 2016;Mordatch and Abbeel, 2018) or implicit (Tian et al, 2020) communication between agents by using an actor-critic policy gradient with a centralized critic for all agents (Foerster et al, 2018). The aim of these protocols is to correctly assign credits so that an agent can deduce its contribution to the team's success.…”