Sufficiency of Markov policies for continuous-time Markov decision processes and solutions to Kolmogorov's forward equation for jump Markov processes

Feinberg, Eugene A.; Mandava, Manasa; Shiryaev, Albert N.

doi:10.1109/cdc.2013.6760792

Cited by 4 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Proof of Theorem 3.1. The following statement is a consequence of Theorem 4.2 of [15], and is the starting point of our reasoning.…”

Section: Proof Of the Main Statementmentioning

confidence: 88%

“…Proof of Theorem 3.1. The following statement is a consequence of Theorem 4.2 of [15], and is the starting point of our reasoning. Lemma 3.1 For each initial state x ∈ S and policy π, there exists a Markov policy ϕ such that…”

Section: Proof Of the Main Statementmentioning

confidence: 88%

“…Proof of Theorem 3.1. The following statement is a consequence of Theorem 4.2 of [16], see also [18], and is the starting point of our reasoning. The above lemma implies that, without loss of generality, we can restrict to the class of Markov policies for problems (I) and (II), i.e.…”

mentioning

confidence: 81%

See 2 more Smart Citations

Note on discounted continuous-time Markov decision processes with a lower bounding function

2017

View full text Add to dashboard Cite

In this paper, we consider the discounted continuous-time Markov decision process (CT-MDP) with a lower bounding function. In this model, the negative part of each cost rate is bounded by the drift function, say w, whereas the positive part is allowed to be arbitrarily unbounded. Our focus is on the existence of a stationary optimal policy for the discounted CTMDP problems out of the more general class. Both constrained and unconstrained problems are considered. Our investigations are based on a useful transformation for nonhomogeneous Markov pure jump processes that has not yet been widely applied to the study of CTMDPs. This technique was not employed in previous literature, but it clarifies the roles of the imposed conditions in a rather transparent way. As a consequence, we withdraw and weaken several conditions commonly imposed in the literature.

show abstract

“…Proof of Theorem 3.1. The following statement is a consequence of Theorem 4.2 of [15], and is the starting point of our reasoning.…”

Section: Proof Of the Main Statementmentioning

confidence: 88%

Section: Proof Of the Main Statementmentioning

confidence: 88%

See 1 more Smart Citation

Note on discounted continuous-time Markov decision processes with a lower bounding function

2017

View full text Add to dashboard Cite

show abstract

“…given in Feller[7, Theorem 4] remains correct for measurable Q-functions.Observe thatP is a transition function if the Q-function q satisfies Assumption 4. For continuous Q-functions satisfying Assumption 1, Feller[7, Theorems 2,5] proved that: (a) for fixed u, x,t the functionP(u, x;t, ·) is a measure on (X, B(X)) such that 0 ≤P(u, x;t, ·) ≤ 1, and (b) for all u, x,t, B the functionP(u, x;t, B) satisfies the Chapman-Kolmogorov equation (1). The proofs remain correct for measurable Q-functions satisfying Assumption 4.…”

mentioning

confidence: 99%

Kolmogorov’s equations for jump Markov processes with unbounded jump rates

2017

Self Cite

View full text Add to dashboard Cite

As is well-known, transition probabilities of jump Markov processes satisfy Kolmogorov's backward and forward equations. In the seminal 1940 paper, William Feller investigated solutions of Kolmogorov's equations for jump Markov processes. Recently the authors solved the problem studied by Feller and showed that the minimal solution of Kolmogorov's backward and forward equations is the transition probability of the corresponding jump Markov process if the transition rate at each state is bounded. This paper presents more general results. For Kolmogorov's backward equation, the sufficient condition for the described property of the minimal solution is that the transition rate at each state is locally integrable, and for Kolmogorov's forward equation the corresponding sufficient condition is that the transition rate at each state is locally bounded.

show abstract

Continuous-Time Markov Chains

Clempner,

Poznyak

2023

Studies in Systems, Decision and Control

View full text Add to dashboard Cite

Sufficiency of Markov policies for continuous-time Markov decision processes and solutions to Kolmogorov's forward equation for jump Markov processes

Cited by 4 publications

References 15 publications

Note on discounted continuous-time Markov decision processes with a lower bounding function

Note on discounted continuous-time Markov decision processes with a lower bounding function

Kolmogorov’s equations for jump Markov processes with unbounded jump rates

Continuous-Time Markov Chains

Contact Info

Product

Resources

About