Strong polynomiality of policy iterations for average-cost MDPs modeling replacement and maintenance problems

Feinberg, Eugene A.; Huang, Jefferson

doi:10.1016/j.orl.2013.02.002

Cited by 9 publications

(6 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…for all x ∈X, there exists a stationary optimal policy for the discounted problem, and a stationary policy is optimal for this problem if and only if (8) holds for all x ∈ X. Corollary 7. Suppose Assumption HT holds with a state ℓ ∈ X that is isolated from X and Assumption WC holds.…”

Section: Hv-ag Transformationmentioning

confidence: 99%

See 1 more Smart Citation

Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted MDPs

Feinberg

Huang

2018

Operations Research Letters

Self Cite

View full text Add to dashboard Cite

This note describes sufficient conditions under which total-cost and average-cost Markov decision processes (MDPs) with general state and action spaces, and with weakly continuous transition probabilities, can be reduced to discounted MDPs. For undiscounted problems, these reductions imply the validity of optimality equations and the existence of stationary optimal policies. The reductions also provide methods for computing optimal policies. The results are applied to a capacitated inventory control problem with fixed costs and lost sales.

show abstract

Section: Hv-ag Transformationmentioning

confidence: 99%

“…One advantage of this approach is that it can be used to apply methods and algorithms developed for discounted MDPs to undiscounted MDPs. [1,8,9].…”

Section: Introductionmentioning

confidence: 99%

Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted MDPs

Feinberg

Huang

2018

Operations Research Letters

Self Cite

View full text Add to dashboard Cite

show abstract

“…Typically, average optimization is a more difficult problem than discounted optimization [7]. PI can need an exponential number of iterations under averageoptimality for stochastic MDPs in the general case [6].…”

Section: Dynamic Programmingmentioning

confidence: 99%

Tuning the Discount Factor in Order to Reach Average Optimality on Deterministic MDPs

Perotto

Vercouter

2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Considering Markovian Decision Processes (MDPs), the meaning of an optimal policy depends on the optimality criterion chosen. The most common approach is to define the optimal policy as the one that maximizes the sum of discounted rewards. The intuitive alternative is to maximize the average reward per step. The former has strong convergence guarantees but suffers from the dependency on a discount factor. The latter has the additional inconvenience of being insensitive to different policies with equivalent average. This paper analyzes the impact of such different criteria on a series of experiments, and then provides a threshold for the discount factor in order to ensure average optimality for discounted-optimal policies in the deterministic case.

show abstract

“…A direct reduction of average‐cost MDPs to discounted ones, which yields sufficient conditions for the existence of stationary average‐cost optimal policies, was established by Ross for MDPs with Borel state space, finite action sets, bounded costs, and a state to which the process will transition from any state under any action with probability at least

α > 0

. This reduction and Ye's results were used by Feinberg and Huang to obtain iteration bounds for average‐cost policy iterations. Gubenko and Štatland showed that a reduction is also possible for MDPs with Borel state space, bounded costs, and compact action sets, if a “minorization” condition, which generalizes Ross's assumption, is satisfied; see also Dynkin and Yushkevich , Chapter 7, §10].…”

Section: Introductionmentioning

confidence: 99%

On the reduction of total‐cost and average‐cost MDPs to discounted MDPs

Feinberg

Huang

2017

Naval Research Logistics

Self Cite

View full text Add to dashboard Cite

This article provides conditions under which total‐cost and average‐cost Markov decision processes (MDPs) can be reduced to discounted ones. Results are given for transient total‐cost MDPs with transition rates whose values may be greater than one, as well as for average‐cost MDPs with transition probabilities satisfying the condition that there is a state such that the expected time to reach it is uniformly bounded for all initial states and stationary policies. In particular, these reductions imply sufficient conditions for the validity of optimality equations and the existence of stationary optimal policies for MDPs with undiscounted total cost and average‐cost criteria. When the state and action sets are finite, these reductions lead to linear programming formulations and complexity estimates for MDPs under the aforementioned criteria.© 2017 Wiley Periodicals, Inc. Naval Research Logistics 66:38–56, 2019

show abstract

Strong polynomiality of policy iterations for average-cost MDPs modeling replacement and maintenance problems

Cited by 9 publications

References 10 publications

Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted MDPs

Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted MDPs

Tuning the Discount Factor in Order to Reach Average Optimality on Deterministic MDPs

On the reduction of total‐cost and average‐cost MDPs to discounted MDPs

Contact Info

Product

Resources

About