Constrained Average Cost Markov Decision Chains

Abstract

A Markov decision chain with denumerable state space incurs two types of costs — for example, an operating cost and a holding cost. The objective is to minimize the expected average operating cost, subject to a constraint on the expected average holding cost. We prove the existence of an optimal constrained randomized stationary policy, for which the two stationary policies differ on at most one state. The examples treated are a packet communication system with reject option and a single-server queue with service rate control.

Joint routing, scheduling and power control providing hard deadline in wireless multihop networks

“…[32] implemented an online algorithm by modifying the value iteration equation that minimizes the average power under an average delay constraint for a single user. Existence of stationary optimal polices for the constrained average-cost Markov decision processes was shown by setting up the problem as a constrained Markov decision process (C-MDP) in ( [2], [13], [38]). Using techniques in Markov decision process (MDP), structural properties of the optimal policy were obtained in [12].…”

Optimal policies for constrained average-cost Markov decision processes

“…It is already known that for Markov decision constraint problems, there exist optimal randomized policies (Beutler and Ross 1985;Borkar 1994;Frid 1972;González-Hernández and Hernández-Lerma 2005;Haviv 1996;Sennott 1991Sennott , 1993. The case of finite, denumerable, and compact state spaces has been widely dealed (Beutler and Ross 1985;Sennott 1993;Borkar 1994;Kurano et al 2000a;Piunovskiy 1993Piunovskiy , 1997Piunovskiy and Khametov 1991;Tanaka 1991;Hu and Yue 2008).…”

“…The case of finite, denumerable, and compact state spaces has been widely dealed (Beutler and Ross 1985;Sennott 1993;Borkar 1994;Kurano et al 2000a;Piunovskiy 1993Piunovskiy , 1997Piunovskiy and Khametov 1991;Tanaka 1991;Hu and Yue 2008). The discounted performance criteria has also already been dealed (Feimberg and Shwartz 1996;González-Hernández and Hernández-Lerma 2005;Hernández-Lerma and González-Hernández 2000;and Sennott 1991).…”

The Linear Program approach in multi-chain Markov Decision Processes revisited

“…Our derivation is based on a Lagrange formulation which generalizes the one used in the case of a single constraint [3] (later extended in [141 [15] to the countable state space) and is related to the approach used by Borkar in his book [5]. Our new derivation seems to be more natural and straightforward than the previous method [9], where the paradigm was to introduce the LP and then to prove that the optimal policy and optimal value of the control problem are appropriately related to the optimal solution and value of the LP.…”

“…We then show for the multi-constrained Markov Decision Process that the Linear Program suggested in [9] can be obtained from an equivalent unconstrained Lagrange formulation of the control problem. This shows the connection between the Linear Program approach and the Lagrange approach, that was previously used only for the case of a single constraint [3,14,15]. …”