Partial and Conditional Expectations in Markov Decision Processes with Integer Weights

Abstract

The paper addresses two variants of the stochastic shortest path problem ("optimize the accumulated weight until reaching a goal state") in Markov decision processes (MDPs) with integer weights. The first variant optimizes partial expected accumulated weights, where paths not leading to a goal state are assigned weight 0, while the second variant considers conditional expected accumulated weights, where the probability mass is redistributed to paths reaching the goal. Both variants constitute useful approaches to the analysis of systems without guarantees on the occurrence of an event of interest (reaching a goal state), but have only been studied in structures with non-negative weights. Our main results are as follows. There are polynomial-time algorithms to check the finiteness of the supremum of the partial or conditional expectations in MDPs with arbitrary integer weights. If finite, then optimal weight-based deterministic schedulers exist. In contrast to the setting of non-negative weights, optimal schedulers can need infinite memory and their value can be irrational. However, the optimal value can be approximated up to an absolute error of ǫ in time exponential in the size of the MDP and polynomial in log(1/ǫ).

Probabilistic causes in Markov chains

“…Fig. 6: The DTMCs N i for i = 0, 1 by reduction to [35]. Nevertheless, in the remainder of this section we argue that computing a pexpcost-minimal p-cause is computationally hard, in contrast to expcost (cf.…”

“…Proof sketch. For the pseudo-polynomial time bound we apply the techniques from [35] to optimize the partial expected cost in MDPs to the p-causal MDP C p (M ). It is shown in [35] that there is an optimal scheduler whose decision depends only on the current state and accumulated weight and that such a scheduler and its partial expectation can be computed in pseudo-polynomial time.…”

“…For the pseudo-polynomial time bound we apply the techniques from [35] to optimize the partial expected cost in MDPs to the p-causal MDP C p (M ). It is shown in [35] that there is an optimal scheduler whose decision depends only on the current state and accumulated weight and that such a scheduler and its partial expectation can be computed in pseudo-polynomial time. It is further shown that a rational number K can be computed in polynomial time such that for accumulated weights above K, an optimal scheduler has to minimize the probability to reach error .…”

“…The minimal partial expectation is PE min = inf S PE S . It is known that there is a scheduler obtaining the minimal partial expectation [35]. Then a p-cause Π and the corresponding scheduler S Π satisfy pexpcost(Π) = PE SΠ .…”

“…More precisely, the results for the expected cost rely on connections to the stochastic shortest path problem (SSP) studied in [3]. The pseudo-polynomial algorithm for partial expected costs on non-negative, accumulated weights uses partial expectations in Markov decision processes [35]. The PP-hardness result is proved by reduction from the cost problem for acyclic Markov chains stated in [22].…”