Finite-Memory Strategies in POMDPs with Long-Run Average Objectives

Abstract

Partially observable Markov decision processes (POMDPs) are standard models for dynamic systems with probabilistic and nondeterministic behaviour in uncertain environments. We prove that in POMDPs with long-run average objective, the decision maker has approximately optimal strategies with finite memory. This implies notably that approximating the long-run value is recursively enumerable, as well as a weak continuity property of the value with respect to the transition function.

History-dependent Evaluations in Partially Observable Markov Decision Process

“…The key point of the proof is the existence of a pure strategy with finite memory that is εoptimal in the problem with lim inf evaluation, proved in [7]. As highlighted before, the lim inf value v ∞ (x 1 ) has been shown to be equal to v * (x 1 ).…”

“…The supremum can be taken over pure strategies as a direct consequence of Theorem 5.2 in Feinberg [10]. This value coincides with the asymptotic value v * (x 1 ) [19], and for all ε > 0, there exists an ε-optimal pure strategy with finite memory [7].…”

“…In particular, we reformulate the payoff as a function of the beliefs and introduce the notion of invariant measure that is the key element of our proof. Second, relying on the existence of a finite-memory strategy that is ε-optimal for the lim inf evaluation [7], we prove that the decision-maker can guarantee v * (x 1 ) − ε in a uniform sense in the POMDP. In order to do so, we will use that v * (x 1 ) = v ∞ (x 1 ).…”

“…Our proof borrows several ingredients from the three papers [19,14,7]. In the sketch of proof below, we emphasize the differences and common points with these works.…”

“…Lower bound Let us prove that the decision-maker can guarantee v * (x 1 ) in Γ θ (x 1 ), for any θ such that I(θ, x 1 ) is close to 0. We rely on the existence of ε-optimal strategies with finite memory for the lim inf-evaluation, proved in [7]. A crucial point is that under a finite memory strategy σ, the process (state, memory state, action, signal) is a finite Markov chain.…”