A minimization problem of the risk probability in first passage semi-Markov decision processes with loss rates

Abstract

This paper is the first attempt to investigate the risk probability criterion in semi-Markov decision processes with loss rates. The goal is to find an optimal policy with the minimum risk probability that the total loss incurred during a first passage time to some target set exceeds a loss level. First, we establish the optimality equation via a successive approximation technique, and show that the value function is the unique solution to the optimality equation. Second, we give suitable conditions, under which we prove the existence of optimal policies and develop an algorithm for computing -optimal policies. Finally, we apply our main results to a business system. Keywordssemi-Markov decision processes, loss rate, risk probability, first passage time, optimal policy, iteration algorithm MSC(2010) 90C40, 60J27Citation: Huang X X, Zou X L, Guo X P. A minimization problem of the risk probability in first passage semiMarkov decision processes with loss rates.

First passage risk probability optimality for continuous time Markov decision processes

“…It follows from Theorem 1 in [16]. (2) Theorem 3.3 is also called Lyapunov condition, which is weaker than the well known regular condition for SMDPs in [10,12,13,14]. This is because the regular condition for SMDPs means that Q(δ, E | i, a) ≤ 1 − ε, for all (i, a) ∈ K (for some δ and > 0), where Q(δ, E | i, a) is the semi-Markov kernel.…”

“…(ii) The other is first passage risk probability criterion, which usually refers to the probability of the total rewards does not exceed a reward level (profit goal) during a first passage time that the state process first enters a target set. This paper belongs to the second group for MDPs, which have DOI: 10.14736/kyb-2019-1-0114 been discussed in [10,14,18,21,20,27]. More precisely, [21] consider risk minimizing problems in discrete time Markov decisions processes (DTMDPs) with a target set.…”

“…Huang and Guo [10] consider the first passage risk probability problem for semi-Markov decisions processes (SMDPs), and obtain the optimality equation and the existence of optimal policies by using a successive approximation technique. Furthermore, Huang, Zou and Guo [14] investigate the minimum risk probability with loss rates for SMDPs. They establish the optimality equation, give suitable conditions to prove the existence of optimal policies, and develop an algorithm for computing ε-optimal policies.…”

“…A common feature to the risk probability criterion (see, [10,14,20,27,25,16]) is that the decision maker considers the reward levels as well as the system states when making decisions, which is different from the classical expected criterion (see [3,4,6,22,23]) and average criterion (see [4,22,28]) for CTMDPs. Therefore, we can not directly use the results of the classical standard criterions for CTMDPs.…”

“…We will consider in this paper the case when the transition rates are unbounded. To deal with this case, we first use the drift condition (see Theorem 3.3) to ensure the non-explosion of the state processes {x t , t ≥ 0}, which is weaker than the well known regular condition for SMDPs in [10,12,13,14], see Remark 3.4. Furthermore, under some suitable conditions, we not only establish the first passage risk probability optimality equation and show the existence of optimal policies, but also use a value iteration technique to calculate the value function (see Theorem 3.10).…”