Risk Probability Minimization Problems for Continuous-Time Markov Decision Processes on Finite Horizon

Risk probability optimization problem for finite horizon continuous time Markov decision processes with loss rate

“…Most of the earlier studies focus on the reward case, that minimizes the risk probability P π (B r ≤ λ) over all the policies π, where B r denotes the total reward during a given time horizon, λ denote the reward level. It is clear that P π (B r ≤ λ) = 1 − P π (B r > λ), which combined with the conclusions from [13,23] suggests that the risk probability minimization problem P π (B r ≤ λ) in [16] is not equivalent to the minimization problem P π (B r > λ) in this paper. Moreover, in some control models such as economic and financial systems, the controller is often focused on the probability that the total loss incurred over a given time horizon exceeds the initial capacity.…”

“…Risk probability optimality problems for Markov decision processes are first divided into three groups that are based on the hold times of the system state: discrete-time Markov decision processes (DTMDPs) [2,26,28,29,30,31], semi-Markov decision processes (SMDPs) [10,11,12,13,25], and continuous-time Markov decision processes DOI: 10.14736/kyb-2021-2-0272 (CTMDPs) [14,15,16]. Then the second classification is grouped by the risk probability optimization problems with the reward case or the loss case.…”

Optimal Control of Probability on a Target Set for Continuous-Time Markov Chains

“…(6) Remark 3: The optimality criterion in this article is similar to that in [14] on optimal risk probability for first passage models of semi-MDPs, since there is no reward/cost structure in these models. The admissible controls we consider here are stochastic processes adapted to the natural filtration generated by the underlying Markov chain, which are different from the policies studied in [14] as well as some continuous-time MDPs with risk probability criteria, such as [15], [16]. In these works, the policies are only taken at each jump point.…”

“…Under some conditions, it was proved that the value function is a solution to the optimality equation. Following the publication of this work, there were some works on continuous-time MDPs with risk probability criteria, such as [4] and [15]. Bhabak and Saha [4] studied a zero-sum stochastic game for continuous-time Markov chains.…”

“…Under some assumptions, they showed the existence of value of the game and also characterized it as the unique solution of a pair of Shapley equations. Huo and Guo [15] dealt with finite horizon continuous-time MDPs with unbounded transition rates and established the existence and uniqueness of a solution of the corresponding optimality equation. They also proved the existence of a risk probability optimal policy.…”