Optimally Solving Dec-POMDPs as Continuous-State MDPs

Abstract

Decentralized partially observable Markov decision processes (Dec-POMDPs) provide a general model for decision-making under uncertainty in decentralized settings, but are difficult to solve optimally (NEXP-Complete). As a new way of solving these problems, we introduce the idea of transforming a Dec-POMDP into a continuous-state deterministic MDP with a piecewise-linear and convex value function. This approach makes use of the fact that planning can be accomplished in a centralized offline manner, while execution can still be decentralized. This new Dec-POMDP formulation, which we call an occupancy MDP, allows powerful POMDP and continuous-state MDP methods to be used for the first time. To provide scalability, we refine this approach by combining heuristic search and compact representations that exploit the structure present in multi-agent domains, without losing the ability to converge to an optimal solution. In particular, we introduce a feature-based heuristic search value iteration (FB-HSVI) algorithm that relies on feature-based compact representations, point-based updates and efficient action selection. A theoretical analysis demonstrates that FB-HSVI terminates in finite time with an optimal solution. We include an extensive empirical analysis using well-known benchmarks, thereby demonstrating that our approach provides significant scalability improvements compared to the state of the art.

A Concise Introduction to Decentralized POMDPs

“…This intuition is correct [Nayyar et al, 2011, Dibangoye et al, 2013, MacDermed and Isbell, 2013. In particular, it is possible make a reduction to special type of POMDP: a non-observable MDP (a POMDP with just one 'NULL' observation).…”

“…However, it turns out that it is possible to replace the dependence on the past joint policy by a so-called plan-time sufficient statistic: a distribution over histories and states [Oliehoek et al, 2013a, Dibangoye et al, 2013. This is useful, since many past joint policies can potentially map to the same statistic, as indicated in Figure 4.5.…”

Multi-objective decision-theoretic planning

“…Recently, major strides in solving Dec-POMDPs have been made. In particular, it has been shown that there is a reduction from Dec-POMDP to a special type of centralized POMDP called a non-observable Markov decision process (NOMDP) (MacDermed and Isbell, 2013;Nayyar et al, 2013;Dibangoye et al, 2013;Oliehoek and Amato, 2014). This allows POMDP solution methods to be employed in the context of DecPOMDPs.…”

Probabilistic Inference Techniques for Scalable Multiagent Decision Making

“…Recently, a number of approaches have been developed that transform a Dec-POMDP into a continuous-state MDP and then use techniques from the POMDP literature to solve the continuous-state MDP (Dibangoye, Amato, Doniec, & Charpillet, 2013a;Dibangoye, Amato, Buffet, & Charpillet, 2013b). The state in such a continuous MDP reformulation of a Dec-POMDP, also called occupancy state, is the probability distribution over the world state and the history of observations each agent has received.…”