Time-average optimal constrained semi-Markov decision processes

Abstract

Optimal causal policies maximizing the time-average reward over a semi-Markov decision process (SMDP), subject to a hard constraint on a time-average cost, are considered. Rewards and costs depend on the state and action, and contain running as well as switching components. It is supposed that the state space of the SMDP is finite, and the action space compact metric. The policy determines an action at each transition point of the SMDP. Under an accessibility hypothesis, several notions of time average are equivalent. A Lagrange multiplier formulation involving a dynamic programming equation is utilized to relate the constrained optimization to an unconstrained optimization parametrized by the multiplier. This approach leads to a proof for the existence of a semi-simple optimal constrained policy. That is, there is at most one state for which the action is randomized between two possibilities; at all other states, an action is uniquely chosen for each state. Affine forms for the rewards, costs and transition probabilities further reduce the optimal constrained policy to ‘almost bang-bang’ form, in which the optimal policy is not randomized, and is bang-bang except perhaps at one state. Under the same assumptions, one can alternatively find an optimal constrained policy that is strictly bang-bang, but may be randomized at one state. Application is made to flow control of a birth-and-death process (e.g., an M/M/s queue); under certain monotonicity restrictions on the reward and cost structure the preceding results apply, and in addition there is a simple acceptance region.

To Wave or Not to Wave? Order Release Policies for Warehouses with an Automated Sorter

“…The following results are obtained through a straightforward adaptation to the discounted case of the proofs of Lemma 3.1, Theorem 4.3 and Theorem 4.4 from Beutler and Ross (1985) and Corollary 3.5 from Beutler and Ross (1986):…”

Efficient QoS Provisioning for Adaptive Multimedia in Mobile Communication Networks by Reinforcement Learning

“…The Lagrange multiplier formulation relating the constrained optimization to an unconstrained optimization [25], [26] is used in this paper to deal with the handoff dropping constraint. To fit into this formulation, we need to include the history information in our state descriptor.…”

Untitled

“…To deal with the fairness constraint, we use the Lagrange multiplier framework studied in Beutler and Ross (1986). Since the fairness constraint is a past-dependent constraint (the vector R(s n+1 ) depends on the rejection ratios over the past history), to fit into this framework, we need to include this history information into our state descriptor.…”

“…If there exists a non-randomized policy π ω * that solves the Bellman optimality equation associated with reward function (29), and in the mean time, achieves the equality in (27), then Beutler and Ross (1986) shows that π ω * is the constrained optimal policy.…”