Centralized Training with Hybrid Execution in Multi-Agent Reinforcement Learning

Abstract

We introduce hybrid execution in multi-agent reinforcement learning (MARL), a new paradigm in which agents aim to successfully perform cooperative tasks with any communication level at execution time by taking advantage of informationsharing among the agents. Under hybrid execution, the communication level can range from a setting in which no communication is allowed between agents (fully decentralized), to a setting featuring full communication (fully centralized). To formalize our setting, we define a new class of multi-agent partially observable Markov decision processes (POMDPs) that we name hybrid-POMDPs, which explicitly models a communication process between the agents. We contribute MARO, an approach that combines an autoregressive predictive model to estimate missing agents' observations, and a dropout-based RL training scheme that simulates different communication levels during the centralized training phase. We evaluate MARO on standard scenarios and extensions of previous benchmarks tailored to emphasize the negative impact of partial observability in MARL. Experimental results show that our method consistently outperforms baselines, allowing agents to act with faulty communication while successfully exploiting shared information.

Field-informed Reinforcement Learning of Collective Tasks with Graph Neural Networks

“…The idea is to learn a policy at simulation time when there is a collective view of the system, and then at runtime use that policy but only with local observations. The typical approach in such cases is based on actor-critic systems [32], [33], [34], [35], where the actor is the distributed policy (with only local information) and the critic is a neural network that takes the overall system state. Mean-field RL [17] is one of such concrete applications of CTDE where the interactions among the population of agents are estimated by considering either the effect of a single agent and the average impact of the entire population or the influence of neighbouring agents.…”