Sufficiency of Markov Policies for Continuous-Time Jump Markov Decision Processes

Abstract

This paper extends to Continuous-Time Jump Markov Decision Processes (CTJMDP) the classic result for Markov Decision Processes stating that, for a given initial state distribution, for every policy there is a (randomized) Markov policy, which can be defined in a natural way, such that at each time instance the marginal distributions of state-action pairs for these two policies coincide. It is shown in this paper that this equality takes place for a CTJMDP if the corresponding Markov policy defines a nonexplosive jump Markov process. If this Markov process is explosive, then at each time instance the marginal probability, that a state-action pair belongs to a measurable set of state-action pairs, is not greater for the described Markov policy than the same probability for the original policy. These results are used in this paper to prove that for expected discounted total costs and for average costs per unit time, for a given initial state distribution, for each policy for a CTJMDP the described a Markov policy has the same or better performance.

Kolmogorov's Equations for Jump Markov Processes and their Applications to Control Problems

“…In this case it is also true that V α (γ, ϕ) ≤ V α (γ, ϕ) for a Markov policy ϕ satisfying (4.6), and the equality takes place if P ϕ γ (t, X) = 1 for all t ∈ R + ; see [10,Theorem 5]. As shown in [10, Example 1], this may not be true if the cost function C also depends on an action chosen at jump epochs.…”

“…As shown in [10, Example 1], this may not be true if the cost function C also depends on an action chosen at jump epochs. Other criteria and additional results can be found in [10,Section 6].…”

“…As follows from the definition provided above, a Markov policy φ is a transition probability φ(da|x, t) from (R + × X, B(R + × X)) to (A, B(A)) such that φ(A(x)|x, t) = 1 for all x ∈ X and for all t ∈ R + . The following lemma combines Lemma 1 and Corollary 1 from [10].…”

“…This paper describes the structure of solutions of Kolmogorov's backward and forward equations for jump Markov processes and applications of these results to continuous-time jump Markov decision processes (CTJMDPs). It is based on our joint publications [8,9,10] with Manasa Mandava .…”

“…In [9] we extended the results from [8] to more general transition intensities; see Assumptions 2.1-2.4 below. In [10] we studied applications to CTJMDPs.…”