The complexity of Policy Iteration is exponential for discounted Markov Decision Processes

Hollanders, Romain; Delvenne, Jean‐Charles; Jungers, Raphaël M.

doi:10.1109/cdc.2012.6426485

Cited by 14 publications

(10 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As pointed out in Scherrer et al (2016), the lower-bound complexity of PI is considered an open problem, at least in the most general MDP formulation. Lower-bounds have been derived in specific cases only, such as deterministic MDPs (Hansen & Zwick, 2010), total reward criterion (Fearnley, 2010) or high discount factor (Hollanders et al, 2012). Even though we did not intend to directly address this open question, our lower bound result seems to be a contribution on its own to the general theory of non-delayed MDPs.…”

Section: Mdps With Delay: a Degradation Examplementioning

confidence: 98%

Acting in Delayed Environments with Non-Stationary Markov Policies

Derman,

Dalal,

Mannor

2021

Preprint

View full text Add to dashboard Cite

The standard Markov Decision Process (MDP) formulation hinges on the assumption that an action is executed immediately after it was chosen. However, assuming it is often unrealistic and can lead to catastrophic failures in applications such as robotic manipulation, cloud computing, and finance. We introduce a framework for learning and planning in MDPs where the decision-maker commits actions that are executed with a delay of m steps. The brute-force state augmentation baseline where the state is concatenated to the last m committed actions suffers from an exponential complexity in m, as we show for policy iteration. We then prove that with execution delay, Markov policies in the original state-space are sufficient for attaining maximal reward, but need to be non-stationary. As for stationary Markov policies, we show they are sub-optimal in general. Consequently, we devise a nonstationary Q-learning style model-based algorithm that solves delayed execution tasks without resorting to state-augmentation. Experiments on tabular, physical, and Atari domains reveal that it converges quickly to high performance even for substantial delays, while standard approaches that either ignore the delay or rely on state-augmentation struggle or fail due to divergence. The code is available at https://github.com/galdl/rl_delay_basic.git.

show abstract

Section: Mdps With Delay: a Degradation Examplementioning

confidence: 98%

Acting in Delayed Environments with Non-Stationary Markov Policies

Derman,

Dalal,

Mannor

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In particular, even if the time-complexity of the value iteration (VI) algorithm for convergence in terms of the number of iterations is polynomial in |X|, |A|, 1/(1 − γ ), and the size of representing the inputs R and P, the dependence on 1/(1 − γ ) is a major drawback (Blondel & Tsitsiklis, 2000). On the other hand, PI's time-complexity for convergence is known to be exponential in general (Hollanders, Delvenne, & Jungers, 2012) even if it is strongly polynomial when γ is fixed (Ye, 2011). Note that the per-iteration computational complexity of VI is O(|A||X | 2 ) and that of PI is O(|X | 3 + |A||X| 2 ).…”

Section: Introductionmentioning

confidence: 95%

Random search for constrained Markov decision processes with multi-policy improvement

Chang

2015

Automatica

View full text Add to dashboard Cite

“…Even tighter upper bounds (still exponential in n) have been shown for k = 2 [6]. Interestingly, the only lower bounds that have been shown for PI are either for the special case of k = 2 [8] [9] or when k is related to n [10] [11]. We contribute lower bounds for arbitrary n ≥ 2, k ≥ 2.…”

Section: Introductionmentioning

confidence: 97%

Lower Bounds for Policy Iteration on Multi-action MDPs

Kumar¹,

Consul²,

Dedhia³

et al. 2020

Preprint

View full text Add to dashboard Cite

Policy Iteration (PI) is a classical family of algorithms to compute an optimal policy for any given Markov Decision Problem (MDP). The basic idea in PI is to begin with some initial policy and to repeatedly update the policy to one from an improving set, until an optimal policy is reached. Different variants of PI result from the (switching) rule used for improvement. An important theoretical question is how many iterations a specified PI variant will take to terminate as a function of the number of states n and the number of actions k in the input MDP. While there has been considerable progress towards upper-bounding this number, there are fewer results on lower bounds. In particular, existing lower bounds primarily focus on the special case of k = 2 actions. We devise lower bounds for k ≥ 3. Our main result is that a particular variant of PI can take Ω(k n/2 ) iterations to terminate. We also generalise existing constructions on 2-action MDPs to scale lower bounds by a factor of k for some common deterministic variants of PI, and by log(k) for corresponding randomised variants.

show abstract

The complexity of Policy Iteration is exponential for discounted Markov Decision Processes

Cited by 14 publications

References 17 publications

Acting in Delayed Environments with Non-Stationary Markov Policies

Acting in Delayed Environments with Non-Stationary Markov Policies

Random search for constrained Markov decision processes with multi-policy improvement

Lower Bounds for Policy Iteration on Multi-action MDPs

Contact Info

Product

Resources

About