Optimal and Approximate Q-value Functions for Decentralized POMDPs

Oliehoek, Frans A.; Spaan, Matthijs T. J.; Vlassis, Nikos

doi:10.1613/jair.2447

Cited by 275 publications

(203 citation statements)

References 64 publications

Supporting

Mentioning

200

Contrasting

Order By: Relevance

“…Each individual agent considers itself part of a hypothetical centralized "team-agent" that has joint control of all the agents that are included in the team and optimizes the joint reward of that team (Sugden 2003; Bratman 2014). Planning under this approach combines all the agents in a game into a single agent and finds a joint plan which optimizes that group objective (De Cote and Littman 2008;Oliehoek, Spaan, and Vlassis 2008;Kleiman-Weiner et al 2016). If J is the set of agents that have joined together as a team, their joint-plan can be characterized as:…”

Section: Operator Composition (Replace)mentioning

confidence: 99%

Theory of Minds: Understanding Behavior in Groups through Inverse Planning

Shum

Kleiman-Weiner

Littman

et al. 2019

AAAI

View full text Add to dashboard Cite

Human social behavior is structured by relationships. We form teams, groups, tribes, and alliances at all scales of human life. These structures guide multi-agent cooperation and competition, but when we observe others these underlying relationships are typically unobservable and hence must be inferred. Humans make these inferences intuitively and flexibly, often making rapid generalizations about the latent relationships that underlie behavior from just sparse and noisy observations. Rapid and accurate inferences are important for determining who to cooperate with, who to compete with, and how to cooperate in order to compete. Towards the goal of building machine-learning algorithms with human-like social intelligence, we develop a generative model of multiagent action understanding based on a novel representation for these latent relationships called Composable Team Hierarchies (CTH). This representation is grounded in the formalism of stochastic games and multi-agent reinforcement learning. We use CTH as a target for Bayesian inference yielding a new algorithm for understanding behavior in groups that can both infer hidden relationships as well as predict future actions for multiple agents interacting together. Our algorithm rapidly recovers an underlying causal model of how agents relate in spatial stochastic games from just a few observations. The patterns of inference made by this algorithm closely correspond with human judgments and the algorithm makes the same rapid generalizations that people do.

show abstract

Section: Operator Composition (Replace)mentioning

confidence: 99%

Theory of Minds: Understanding Behavior in Groups through Inverse Planning

Shum

Kleiman-Weiner

Littman

et al. 2019

AAAI

View full text Add to dashboard Cite

show abstract

“…Emery-Montemerlo et al (2004) approximated a cooperative POSG by a series of Bayesian games. Another cooperative POSG, called a decentralized partially observable Markov decision process (DEC-POMDP), has also been extensively studied by Becker et al (2004), Bernstein et al (2005), Seuken and Zilberstein (2007), and Oliehoek et al (2008). A survey of the DEC-POMDP can be found in Oliehoek (2012).…”

Section: The Partially Observable Stochastic Gamementioning

confidence: 99%

A leader–follower partially observed, multiobjective Markov game

2015

View full text Add to dashboard Cite

The intent of this research is to generate a set of non-dominated finite-memory policies from which one of two agents (the leader) can select a most preferred policy to control a dynamic system that is also affected by the control decisions of the other agent (the follower). The problem is described by an infinite horizon total discounted reward, partially observed Markov game (POMG). For each candidate finite-memory leader policy, we assume the follower, fully aware of the leader policy, determines a (perfect memory) policy that optimizes the follower's (scalar) criterion. The leader-follower assumption allows the POMG to be transformed into a specially structured, partially observed Markov decision process that we use to determine the follower's best response policy for a given leader policy. We then approximate the follower's policy by a finite-memory policy. Each agent's policy assumes that the agent knows its current and recent state values, its recent actions, and the current and recent possibly inaccurate observations of the other agent's state. For each leader/follower policy pair, we determine the values of the leader's criteria. We use a multiobjective genetic algorithm to create the next generation of leader policies based on the values of the leader criteria for each leader/follower policy pair in the current generation. Based on this information for the final generation of policies, we determine the set of non-dominated leader policies. We present an example that illustrates how these results can be used to support a manager of a liquid egg production process (the leader) in selecting a sequence of actions to maximize expected process productivity while mitigating the risk due to an attacker (the follower) who seeks to contaminate the process with a chemical or biological toxin.

show abstract

“…When communication delays are limited and potentially stochastic, the problem can be modeled as a multiagent POMDP with delayed communication [33,54]. Finally, when no communication channel is present, the problem can be modeled as a decentralized POMDP [6,34]. Extending the POMDP-IR framework to any of these multiagent models is promising.…”

Section: Future Workmentioning

confidence: 99%

Decision-theoretic planning under uncertainty with information rewards for active cooperative perception

Spaan

Veiga

Lima

2014

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

Partially observable Markov decision processes (POMDPs) provide a principled framework for modeling an agent's decision-making problem when the agent needs to consider noisy state estimates. POMDP policies take into account an action's influence on the environment as well as the potential information gain. This is a crucial feature for robotic agents which generally have to consider the effect of actions on sensing. However, building POMDP models which reward information gain directly is not straightforward, but is important in domains such as robot-assisted surveillance in which the value of information is hard to quantify. Common techniques for uncertainty reduction such as expected entropy minimization lead to non-standard POMDPs that are hard to solve. We present the POMDP with Information Rewards (POMDP-IR) modeling framework, which rewards an agent for reaching a certain level of belief regarding a state feature. By remaining in the standard POMDP setting we can exploit many known results as well as successful approximate algorithms. We demonstrate our ideas in a toy problem as well as in real robot-assisted surveillance, showcasing their use for active cooperative perception scenarios. Finally, our experiments show that the POMDP-IR framework compares favorably with a related approach on benchmark domains.

show abstract

Optimal and Approximate Q-value Functions for Decentralized POMDPs

Cited by 275 publications

References 64 publications

Theory of Minds: Understanding Behavior in Groups through Inverse Planning

Theory of Minds: Understanding Behavior in Groups through Inverse Planning

A leader–follower partially observed, multiobjective Markov game

Decision-theoretic planning under uncertainty with information rewards for active cooperative perception

Contact Info

Product

Resources

About