Complexity of finite-horizon Markov decision process problems

Abstract

Controlled stochastic systems occur in science engineering, manufacturing, social sciences, and many other cntexts. If the systems is modeled as a Markov decision process (MDP) and will run ad infinitum , the optimal control policy can be computed in polynomial time using linear programming. The problems considered here assume that the time that the process will run is finite, and based on the size of the input. There are mny factors that compound the complexity of computing the optimal policy. For instance, there are many factors that compound the complexity of this computation. For instance, if the controller does not have complete information about the state of the system, or if the system is represented in some very succint manner, the optimal policy is provably not computable in time polynomial in the size of the input. We analyze the computational complexity of evaluating policies and of determining whether a sufficiently good policy exists for a MDP, based on a number of confounding factors, including the observability of the system state; the succinctness of the representation; the type of policy; even the number of actions relative to the number of states. In almost every case, we show that the decision problem is complete for some known complexity class. Some of these results are familiar from work by Papadimitriou and Tsitsiklis and others, but some, such as our PL-completeness proofs, are surprising. We include proofs of completeness for natural problems in the as yet little-studied classes NP PP .

Upper Confidence Trees with Short Term Partial Information

“…The crucial assumption is here the presence of nodes with simultaneous actions. Without such nodes, the solving of such games is well known (see [11,13,12] for more on this).…”

“…exponential time, exponential space, doublyexponential time) for the fully observable, no observation, and partially observable case respectivelly for the criterion of deciding whether a 100% winning strategy exists 3 . With exponential horizon, the complexities decrease to EXP, NEXP, EXPSPACE respectively [11]. -With two players without random part, the problem of approximating the best winning probability that can be achieved regardless of the opponent strategy is undecidable [14] by reduction to the one-player randomized case above in the no observation case; the best complexity upper bounds for bounded horizon are 3EXP (for exponential horizon) and 2EXP (for polynomial horizon).…”

“…The existence of strategies winning with probability 1, independently of the opponent, is decidable for 2 players, even in partially observable environments (see [6], showing that this is not true if we have a team of 2 players against a third player). -The fully observable setting is always decidable, with complexity reduced by far in the case of limited horizon; see [13,11] for more on this for the case in which we consider the existence of strategies winning with probability 1 and [14] for the choice of optimal moves.…”

Approximate Receding Horizon Approach for Markov Decision Processes: Average Award Case

“…Unfortunately, obtaining the true H-horizon optimal value is often difficult, e.g., due to the large state space (see, e.g., [29] for a discussion of the complexity of solving finitehorizon MDPs). Motivated by this, we study an approximate receding horizon control that uses an approximate value function as an approximate solution of V * H−1 for some H < ∞.…”

Probabilistic Inference Techniques for Scalable Multiagent Decision Making

“…In terms of computational complexity, optimally solving a finitehorizon Dec-POMDP is NEXP-Complete (Bernstein et al, 2002). In contrast, finite-horizon POMDPs are PSPACE-complete (Mundhenk, Goldsmith, Lusena, & Allender, 2000), a strictly lower complexity class that highlights the difficulty of solving Dec-POMDPs.…”