On the reduction of total‐cost and average‐cost MDPs to discounted MDPs

Abstract

This article provides conditions under which total‐cost and average‐cost Markov decision processes (MDPs) can be reduced to discounted ones. Results are given for transient total‐cost MDPs with transition rates whose values may be greater than one, as well as for average‐cost MDPs with transition probabilities satisfying the condition that there is a state such that the expected time to reach it is uniformly bounded for all initial states and stationary policies. In particular, these reductions imply sufficient conditions for the validity of optimality equations and the existence of stationary optimal policies for MDPs with undiscounted total cost and average‐cost criteria. When the state and action sets are finite, these reductions lead to linear programming formulations and complexity estimates for MDPs under the aforementioned criteria.© 2017 Wiley Periodicals, Inc. Naval Research Logistics 66:38–56, 2019

Sufficiency of Deterministic Policies for Atomless Discounted and Uniformly Absorbing MDPs with Multiple Criteria

“…Of course, the transformation of a discounted MDP into an absorbing one is trivial. However, under certain conditions it is also possible to transform an absorbing MDP into a discounted one; see Feinberg and Huang [14,15].…”

Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted MDPs

“…For x ∈X and π ∈ Π, letṽ π (x), be the expected total discounted cost for the new model, and let v(x) := inf π∈Πṽ π (x). It is well-known (see e.g., [9])…”

“…In this section, we present the HV transformation [9], which is based on ideas due to Alan Hoffman and A. F. Veinott [18]. A point s is isolated from a metric space S, if there exists an ǫ > 0 such that the distance between s and any element of S is larger than ǫ.…”

“…In [9], conditions were given under which undiscounted MDPs with general state and action spaces can be reduced to discounted ones. These conditions include the assumption that the transition probabilities are setwise continuous.…”

“…However, for many models of interest, such as those arising in inventory control [7], the transition probabilities are only weakly continuous. In this paper, we provide conditions under which the reductions in [9] lead to optimality results for undiscounted MDPs with weakly continuous transition kernels. In particular, under these conditions the discounted MDPs to which the undiscounted MDPs are reduced have weakly continuous transition probabilities.…”