The Expected Total Cost Criterion for Markov Decision Processes under Constraints: A Convex Analytic Approach

Abstract

This paper deals with discrete-time Markov decision processes (MDPs) under constraints where all the objectives have the same form of expected total cost over the infinite time horizon. The existence of an optimal control policy is discussed by using the convex analytic approach. We work under the assumptions that the state and action spaces are general Borel spaces, and that the model is nonnegative, semicontinuous, and there exists an admissible solution with finite cost for the associated linear program. It is worth noting that, in contrast to the classical results in the literature, our hypotheses do not require the MDP to be transient or absorbing. Our first result ensures the existence of an optimal solution to the linear program given by an occupation measure of the process generated by a randomized stationary policy. Moreover, it is shown that this randomized stationary policy provides an optimal solution to this Markov control problem. As a consequence, these results imply that the set of randomized stationary policies is a sufficient set for this optimal control problem. Finally, our last main result states that all optimal solutions of the linear program coincide on a special set with an optimal occupation measure generated by a randomized stationary policy. Several examples are presented to illustrate some theoretical issues and the possible applications of the results developed in the paper.

Optimal Impulse Control of Dynamical Systems

“…is called to be of finite value. It can be shown as in [9,Thm.3 Appendix 4] or [12]. It can be shown as in the proof of [9, Thm.3.3, Cor.3.1] that the stationary strategy π µ defined by…”

“…According to [9,Thm.4.1], for a constrained total cost MDP with Borel state space X ∆ , Borel action space B, transition probability Q, and positive cost functions { C j } J j=0 , if the model is semicontinuous, then, provided that there exists a feasible strategy with finite value, there is an optimal stationary strategy. Here the model is called semicontinuous if its action space B is compact, { C j } J j=0 are all lower semicontinuous, and Q is continuous, i.e., for each bounded continuous function f on X ∆ ,…”

“…Throughout this paper, we fix this measurable mapping f * . It can be shown as in [9,Prop.3.2] that Q(V c |x, f * (x)) = 0 for each x ∈ V.…”

“…The first possible definition comes from the occupation measures of the MDP corresponding to the impulse control problem. Based on results in the recent work [9] for MDPs, we establish a linear programming approach for the impulse control problem, and obtain, under very general and natural conditions, the existence of an optimal (possibly randomized) stationary policy. This linear programming approach will be referred to as the first because it is based on the first definition of occupation measures.…”

“…This linear programming approach will be referred to as the first because it is based on the first definition of occupation measures. Here the problematic issue is that while the MDP model in [9] is required to be semicontinuous, the induced MDP from our impulse control problem does not have a continuous transition kernel. To get over this difficulty, we use the following trick: the state space can be extended in such a way that one can introduce a suitable topology on it, with respect to which, the resulting MDP model becomes semicontinuous.…”