Average Cost Optimal Stationary Policies in Infinite State Markov Decision Processes with Unbounded Costs

Abstract

We deal with infinite state Markov decision processes with unbounded costs. Three simple conditions, based on the optimal discounted value function, guarantee the existence of an expected average cost optimal stationary policy. Sufficient conditions are the existence of a distinguished state of smallest discounted value and a single stationary policy inducing an irreducible, ergodic Markov chain for which the average cost of a first passage from any state to the distinguished state is finite. A result to verify this is also given. Two examples illustrate the ease of applying the criteria.

Quantitative Models for Reverse Logistics

“…Moreover, as discussed above, the model is equivalent to a discrete time model by considering the system state at transition epochs. For the discrete model general results on average cost Markov decision problems (see, e.g., Sennott, 1989) assure the existence of a stationary average cost optimal policy. Average cost optimality of an (s, Q)-policy follows since any stationary policy based on the inventory position in the above model is equal to an (s, Q)-policy up to a transient phase.…”

“…To this end, we exploit general theory of Markov decision processes that has been well developed in the past two decades. In particular, we make use of Sennott's results on infinite state Markov decision processes with unbounded costs (Sennott, 1989).…”

“…Sennott (1989) shows convergence of successive approximation to the infinite horizon discounted cost function under the condition that V α (y) is finite for all y and α. Moreover, this condition is shown to hold if there exists a stationary policy inducing an ergodic Markov chain and yielding finite average costs in steady state.…”

“…Sufficient conditions have been given by Sennott (1989) for general Markov decision processes. We show that these conditions hold in our return flow model.…”

A survey of Markov decision models for control of networks of queues

“…Techniques for deriving the former from the latter are now well developed. Recent results specifically motivated by control of queues may be found in Borkar [8][9][10], Weber and Stidham [67], Cavazos-Cadena [12,13], Sennott [54,55]. For a survey, see Arapostathis et al [2].…”