Perseus: Randomized Point-based Value Iteration for POMDPs

Abstract

Partially observable Markov decision processes (POMDPs) form an attractive and principled framework for agent planning under uncertainty. Point-based approximate techniques for POMDPs compute a policy based on a finite set of points collected in advance from the agents belief space. We present a randomized point-based value iteration algorithm called Perseus. The algorithm performs approximate value backup stages, ensuring that in each backup stage the value of each point in the belief set is improved; the key observation is that a single backup may improve the value of many belief points. Contrary to other point-based methods, Perseus backs up only a (randomly selected) subset of points in the belief set, sufficient for improving the value of each belief point in the set. We show how the same idea can be extended to dealing with continuous action spaces. Experimental results show the potential of Perseus in large scale POMDP problems

Efficient Planning in Large POMDPs through Policy Graph Based Factorized Approximations

“…However, such algorithms can handle only problems with few states. Point based value iteration algorithms do not compute a solution for all beliefs, but either sample a set of beliefs (as in Perseus [3]) before the main algorithm or select beliefs during planning (as in HSVI [4]). 'Value iteration' refers to updating the value for a belief using a form of the Bellman equation (2).…”

“…Similarly to Perseus [3], FBVP starts by sampling a set of beliefs B before the actual planning. The main planning algorithm (Algorithm 1) takes the belief set B as input and produces a policy graph that can be used for decision making during online operation.…”

“…We consider POMDPs with discrete state spaces; if the state is described by N variables, the number of states is at worst exponential in N and the number of state transitions is at worst exponential in 2N . To combat computational intractability, many planning algorithms have been introduced with various approaches for improving efficiency [3][4][5]; we describe several approaches in Section 2.1. Overall, however, computational intractability remains a large problem which limits current applicability of POMDPs.…”

“…We compare the performance of our method to four existing POMDP solutions: two traditional approaches (Perseus [3] and HSVI [4]) and two methods designed for large problems (Symbolic Perseus [5] and Truncated Krylov Iteration combined with Perseus [5]). We compare the methods on four POMDP benchmark problems of scalable size, including two new benchmarks introduced here: the Uncertain RockSample problem, which is a more difficult variant of the traditional RockSample benchmark, and Spectrum Access which is adapted from a cognitive radio application and is described further below.…”

Controller Compilation and Compression for Resource Constrained Applications

“…Policy optimization algorithms can be classified in two broad categories: i) offline techniques that pre-compute a policy before the start of the execution [14,15,16] and ii) online techniques that perform all their computation at run time by searching for the best action to execute after receiving each observation [3]. Online techniques can take advantage of the history so far to focus their computation only on the current belief.…”