Finite horizon risk-sensitive continuous-time Markov decision processes with unbounded transition and cost rates

Abstract

We consider a risk-sensitive continuous-time Markov decision process over a finite time duration. Under the conditions that can be satisfied by unbounded transition and cost rates, we show the existence of an optimal policy, and the existence and uniqueness of the solution to the optimality equation out of a class of possibly unbounded functions, to which the Feynman-Kac formula was also justified to hold.

Risk-sensitive average continuous-time Markov decision processes with unbounded transition and cost rates

“…To prove the existence of risk-sensitive average optimal policies, we need to develop some preliminary facts concerning risk-sensitive finite-horizon CTMDPs, some of which are from [9] and some of which are new.…”

“…Since the transition and cost rates may be unbounded, we need the following conditions from [9,12,15,23] to guarantee the non-explosion of {x t , t ≥ 0} and the finiteness of J(i, π). Assumption 3.1.…”

“…To characterize risk-sensitive CTMDPs over the class of all history-dependent policies, we need an extension of the Feymanm-Kac formula in [9] from the Markovian case to a more general case, which is based on the following conditions.…”

“…In comparison, there has been limited work on risk-sensitive continuous-time Markov decision processes (CTMDPs). For background on risk-sensitive CTMDPs, see [7,11,22] for the risk-sensitive (infinite-horizon) discounted criterion, [7,22,23,25,26,27] for the risk-sensitive (long-run) average criterion, [7,9,14,17,18,23,24] for the risk-sensitive finite-horizon criterion, and [8,28] for the risk-sensitive total cost criterion. In this paper we will further consider the risksensitive average criterion for CTMDPs studied in [7,22,23,25,26,27].…”

“…First, under the milder conditions from [9,12,14], as in the arguments of [7,23] we derive the multiplicative dynamic programming principle and obtain some interesting results, such as the existence and uniqueness of a solution to the optimality equation and properties of the value function for risk-sensitive finite-horizon CTMDPs with unbounded transition and cost rates (Theorem 3.1 and Lemma 3.2). Second, using the multiplicative dynamic programming principle and properties developed here for risk-sensitive finite-horizon CTMDPs, by the method in [7,23] we obtain the existence of a solution to the RS-AOE and its characterization for risk-sensitive average CTMDPs (Theorems 4.1 and 6.2).…”