Integrating Decision Sharing with Prediction in Decentralized Planning for Multi-Agent Coordination under Uncertainty

Abstract

The performance of decentralized multi-agent systems tends to benefit from information sharing and its effective utilization. However, too much or unnecessary sharing may hinder the performance due to the delay, instability and additional overhead of communications. Aiming to a satisfiable coordination performance, one would prefer the cost of communications as less as possible. In this paper, we propose an approach for improving the sharing utilization by integrating information sharing with prediction in decentralized planning. We present a novel planning algorithm by combining decision sharing and prediction based on decentralized Monte Carlo Tree Search called Dec-MCTS-SP. Each agent grows a search tree guided by the rewards calculated by the joint actions, which can not only be sampled from the shared probability distributions over action sequences, but also be predicted by a sufficiently-accurate and computationally-cheap heuristics-based method. Besides, several policies including sparse and discounted UCT and DIY-bonus are leveraged for performance improvement. We have implemented Dec-MCTS-SP in the case study on multi-agent information gathering under threat and uncertainty, which is formulated as Decentralized Partially Observable Markov Decision Process (Dec-POMDP). The factored belief vectors are integrated into Dec-MCTS-SP to handle the uncertainty. Comparing with the random, auction-based algorithm and Dec-MCTS, the evaluation shows that Dec-MCTS-SP can reduce communication cost significantly while still achieving a surprisingly higher coordination performance.

Decentralized MCTS via Learned Teammate Models

“…In this paper, we take a so-called subjective perspective [Oliehoek and Amato, 2016] of the multi-agent scenario, in which the system is modeled from a protagonist agent's view. The simplest approach simply ignores the other agents completely (some-times called 'self-absorbed' [Claes et al, 2015]).…”

“…Such models can be tables, heuristics, finite-state machines, neural networks, other machine learning models, etc. Given that these do not explicitly model other agents, they have been called 'sub-intentional', however, as demonstrated by [Rabinowitz et al, 2018] they can demonstrate complex charac-teristics associated with the 'theory of mind ' [Premack and Woodruff, 1978].…”

“…The authors however assume heuristic (so not always accurate) models of other agents and do not learn their actual policies. [Best et al, 2016] uses parallel MCTS for an active perception task and combines it with communication to solve the coordination problem; [Li et al, 2019] explores similar ideas and combines communication with heuristic teammate models of [Claes et al, 2017]. Contrary to both of these papers, we assume no communication during the execution phase.…”

“…Decentralized planning methods were applied in context of multiplayer computer games [Jaderberg et al, 2019], robot soccer [Aşık and Akın, 2012], intersection control [Vu et al, 2018] and autonomous warehouse control [Claes et al, 2017], to name a few. The essential difficulty of this paradigm lies in solving the coordination problem.…”

“…For instance, consider a relatively simplistic instance of a spatial task allocation problem in which a team of n robotic vacuum cleaners needs to clean a factory floor, as in Figure 1. Assuming that a robot solves its own traveling salesman problem [Lin, 1965] would result in optimal path planning if it was alone in the factory; but collectively it could lead to unnecessary duplication of resources with multiple robots heading to the same littered area. On the other hand, joint optimization of all actions results in an intractable problem, that is not scalable to large networks of agents.…”