Continuous-Time Markov Decision Processes with Exponential Utility

Abstract

In this paper, we consider a continuous-time Markov decision process (CTMDP) in Borel spaces, where the certainty equivalent with respect to the exponential utility of the total undiscounted cost is to be minimized. The cost rate is nonnegative. We establish the optimality equation. Under the compactness-continuity condition, we show the existence of a deterministic stationary optimal policy. We reduce the risk-sensitive CTMDP problem to an equivalent risk-sensitive discrete-time Markov decision process, which is with the same state and action spaces as the original CTMDP. In particular, the value iteration algorithm for the CTMDP problem follows from this reduction. We do not need impose any condition on the growth of the transition and cost rate in the state, and the controlled process could be explosive.to the optimality equation, provided that it satisfies certain conditions. The question of when there exits such a solution to the optimality equation was not discussed in [27]. For the same problem as in [27], this question was only considered in the recent papers [16,32]. In [16] the transition rates were assumed to be bounded; and in [32] the growth of the transition rate was assumed to be bounded by some Lyapunov function. As a consequence, the controlled process in [16,32] is nonexplosive under each policy. In [16,32], the cost rate was assumed to be bounded, and the arguments are based on Dynkin's formula or the Feynman-Kac formula. The author of [32] explained why it was hard to relax this boundedness condition on the cost rate if one follows the same approach as in there, see Section 7 therein. On the other hand, one should note that unbounded transition and cost rates appear in many real-life applications; as a simplest example, consider an M/M/∞ queueing system with the holding cost rate being proportional to the number of enqueued customers.By the way, the CTMDP in [16,27,32] is assumed to be in a denumerable state space. In [16], the infinite horizon discounted and average problems for the risk-sensitive CTMDP were also considered.The present paper also deals with a CTMDP with exponential utility, but is rather different from the aforementioned works [16,27,32] in the following aspects. (a) We consider the problem of minimizing the expectation of the exponential utility of the total undiscounted cost over the infinite time horizon. In the current literature, we are not aware of other work on the infinite horizon total undiscounted cost criterion for the risk-sensitive CTMDP. (b) Our method of attack does not involve the Dynkin's formula or the Feyman-Kac formula, but is based on the reduction of the risk-sensitive CTMDP to a risk-sensitive DTMDP. As an advantage of developing this approach, we do not need bounds on the growth of the transition and cost rates, the controlled process is allowed to be explosive, and the state space is a general Borel space. Such explosive processes are related to the "shattering into dust" phenomenon in physics, see [31]. Our reduction method starts with a risk-sensiti...

On Risk-Sensitive Piecewise Deterministic Markov Decision Processes

“…For ease of reference, we present the relevant notations and facts about the risk-sensitive problem for a DTMDP. The proofs of the presented statements can be found in [16] or [27]. Standard description of a DTMDP can be found in e.g., [17,22].…”

Finite horizon risk-sensitive continuous-time Markov decision processes with unbounded transition and cost rates

“…An early work on this topic seems to be [18], which obtained verification theorems and solved in closed-form meaningful examples of problems over a fixed time duration. In the recent years, there have been reviving interests in risk-sensitive CTMDPs, see [5,6,21] for problems with a finite horizon, [23] for problems over an infinite horizon, [5,16,22] for problems with average criteria, and [2] for an optimal stopping problem with a more general utility function than the exponential one.…”

Risk-sensitive average continuous-time Markov decision processes with unbounded transition and cost rates

“…We denote by the set of all randomized history-dependent policies, by r m the set of all randomized Markov policies, by d m the set of all (deterministic) Markov policies, and by F the set of all stationary policies. Obviously, ⊃ r m ⊃ d m ⊃ F. For any initial distribution γ on S and policy π ∈ , we introduce the unique probability measure P π γ on ( , F) as in [12,13,28]. Let E π γ be its corresponding expectation operator.…”

“…In comparison, there has been limited work on risk-sensitive continuous-time Markov decision processes (CTMDPs). For background on risk-sensitive CTMDPs, see [7,11,22] for the risk-sensitive (infinite-horizon) discounted criterion, [7,22,23,25,26,27] for the risk-sensitive (long-run) average criterion, [7,9,14,17,18,23,24] for the risk-sensitive finite-horizon criterion, and [8,28] for the risk-sensitive total cost criterion. In this paper we will further consider the risksensitive average criterion for CTMDPs studied in [7,22,23,25,26,27].…”