Kolmogorov’s equations for jump Markov processes with unbounded jump rates

Feinberg, Eugene A.; Mandava, Manasa; Shiryaev, Albert N.

doi:10.1007/s10479-017-2538-8

Cited by 12 publications

(19 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Proof. This follows from the reasoning of the proof of Lemma 1(a) in [14] based on the Novikov seperation theorem. ✷…”

Section: Existence Of a Deterministic Stationary Optimal Policymentioning

confidence: 96%

Continuous-Time Markov Decision Processes with Exponential Utility

Zhang¹

2017

SIAM J. Control Optim.

View full text Add to dashboard Cite

In this paper, we consider a continuous-time Markov decision process (CTMDP) in Borel spaces, where the certainty equivalent with respect to the exponential utility of the total undiscounted cost is to be minimized. The cost rate is nonnegative. We establish the optimality equation. Under the compactness-continuity condition, we show the existence of a deterministic stationary optimal policy. We reduce the risk-sensitive CTMDP problem to an equivalent risk-sensitive discrete-time Markov decision process, which is with the same state and action spaces as the original CTMDP. In particular, the value iteration algorithm for the CTMDP problem follows from this reduction. We do not need impose any condition on the growth of the transition and cost rate in the state, and the controlled process could be explosive.to the optimality equation, provided that it satisfies certain conditions. The question of when there exits such a solution to the optimality equation was not discussed in [27]. For the same problem as in [27], this question was only considered in the recent papers [16,32]. In [16] the transition rates were assumed to be bounded; and in [32] the growth of the transition rate was assumed to be bounded by some Lyapunov function. As a consequence, the controlled process in [16,32] is nonexplosive under each policy. In [16,32], the cost rate was assumed to be bounded, and the arguments are based on Dynkin's formula or the Feynman-Kac formula. The author of [32] explained why it was hard to relax this boundedness condition on the cost rate if one follows the same approach as in there, see Section 7 therein. On the other hand, one should note that unbounded transition and cost rates appear in many real-life applications; as a simplest example, consider an M/M/∞ queueing system with the holding cost rate being proportional to the number of enqueued customers.By the way, the CTMDP in [16,27,32] is assumed to be in a denumerable state space. In [16], the infinite horizon discounted and average problems for the risk-sensitive CTMDP were also considered.The present paper also deals with a CTMDP with exponential utility, but is rather different from the aforementioned works [16,27,32] in the following aspects. (a) We consider the problem of minimizing the expectation of the exponential utility of the total undiscounted cost over the infinite time horizon. In the current literature, we are not aware of other work on the infinite horizon total undiscounted cost criterion for the risk-sensitive CTMDP. (b) Our method of attack does not involve the Dynkin's formula or the Feyman-Kac formula, but is based on the reduction of the risk-sensitive CTMDP to a risk-sensitive DTMDP. As an advantage of developing this approach, we do not need bounds on the growth of the transition and cost rates, the controlled process is allowed to be explosive, and the state space is a general Borel space. Such explosive processes are related to the "shattering into dust" phenomenon in physics, see [31]. Our reduction method starts with a risk-sensiti...

show abstract

“…Proof. This follows from the reasoning of the proof of Lemma 1(a) in [14] based on the Novikov seperation theorem. ✷…”

Section: Existence Of a Deterministic Stationary Optimal Policymentioning

confidence: 96%

Continuous-Time Markov Decision Processes with Exponential Utility

Zhang¹

2017

SIAM J. Control Optim.

View full text Add to dashboard Cite

show abstract

“…Several of the statements below do not need the bounding function w in Condition 3 to be continuous. In this connection, we also mention that a Borel measurable function w satisfying the inequality in Condition 1 always exists; see Lemma 1 of [9] and recall (1).…”

Section: Optimality Resultsmentioning

confidence: 99%

On gradual-impulse control of continuous-time Markov decision processes with exponential utility

et al. 2021

View full text Add to dashboard Cite

We consider a gradual-impulse control problem of continuous-time Markov decision processes, where the system performance is measured by the expectation of the exponential utility of the total cost. We show, under natural conditions on the system primitives, the existence of a deterministic stationary optimal policy out of a more general class of policies that allow multiple simultaneous impulses, randomized selection of impulses with random effects, and accumulation of jumps. After characterizing the value function using the optimality equation, we reduce the gradual-impulse control problem to an equivalent simple discrete-time Markov decision process, whose action space is the union of the sets of gradual and impulsive actions.

show abstract

“…Proof of Theorem 3.1. The following statement is a consequence of Theorem 4.2 of [16], see also [18], and is the starting point of our reasoning. The above lemma implies that, without loss of generality, we can restrict to the class of Markov policies for problems (I) and (II), i.e.…”

mentioning

confidence: 79%

Note on discounted continuous-time Markov decision processes with a lower bounding function

2017

View full text Add to dashboard Cite

In this paper, we consider the discounted continuous-time Markov decision process (CT-MDP) with a lower bounding function. In this model, the negative part of each cost rate is bounded by the drift function, say w, whereas the positive part is allowed to be arbitrarily unbounded. Our focus is on the existence of a stationary optimal policy for the discounted CTMDP problems out of the more general class. Both constrained and unconstrained problems are considered. Our investigations are based on a useful transformation for nonhomogeneous Markov pure jump processes that has not yet been widely applied to the study of CTMDPs. This technique was not employed in previous literature, but it clarifies the roles of the imposed conditions in a rather transparent way. As a consequence, we withdraw and weaken several conditions commonly imposed in the literature.

show abstract

Kolmogorov’s equations for jump Markov processes with unbounded jump rates

Cited by 12 publications

References 15 publications

Continuous-Time Markov Decision Processes with Exponential Utility

Continuous-Time Markov Decision Processes with Exponential Utility

On gradual-impulse control of continuous-time Markov decision processes with exponential utility

Note on discounted continuous-time Markov decision processes with a lower bounding function

Contact Info

Product

Resources

About