Decisive Markov Chains

Abstract

Abstract. We consider qualitative and quantitative verification problems for infinitestate Markov chains. We call a Markov chain decisive w.r.t. a given set of target states F if it almost certainly eventually reaches either F or a state from which F can no longer be reached. While all finite Markov chains are trivially decisive (for every set F ), this also holds for many classes of infinite Markov chains.Infinite Markov chains which contain a finite attractor are decisive w.r.t. every set F . In particular, all Markov chains induced by probabilistic lossy channel systems (PLCS) contain a finite attractor and are thus decisive. Furthermore, all globally coarse Markov chains are decisive. The class of globally coarse Markov chains includes, e.g., those induced by probabilistic vector addition systems (PVASS) with upward-closed sets F , and all Markov chains induced by probabilistic noisy Turing machines (PNTM) (a generalization of the noisy Turing machines (NTM) of Asarin and Collins).We consider both safety and liveness problems for decisive Markov chains. Safety: What is the probability that a given set of states F is eventually reached. Liveness: What is the probability that a given set of states is reached infinitely often. There are three variants of these questions. (1) The qualitative problem, i.e., deciding if the probability is one (or zero); (2) the approximate quantitative problem, i.e., computing the probability up-to arbitrary precision; (3) the exact quantitative problem, i.e., computing probabilities exactly.1. We express the qualitative problem in abstract terms for decisive Markov chains, and show an almost complete picture of its decidability for PLCS, PVASS and PNTM.2. We also show that the path enumeration algorithm of Iyer and Narasimha terminates for decisive Markov chains and can thus be used to solve the approximate quantitative safety problem. A modified variant of this algorithm can be used to solve the approximate quantitative liveness problem.3. Finally, we show that the exact probability of (repeatedly) reaching F cannot be effectively expressed (in a uniform way) in Tarski-algebra for either PLCS, PVASS or (P)NTM (unlike for probabilistic pushdown automata).2000 ACM Subject Classification: G3, D2.4, F4.1.

Qualitative Analysis of VASS-Induced MDPs

“…From this inclusion, the previous equality and using (2), we deduce that P(M S , c, σ, ♦ ¬q F ) ≤ i∈N P(M S , c, σ, ∆ c−i ) = 0. Hence, for all c ∈ F , we have P(M S , c, σ, ♦q F ) = 1, which allows us to conclude that…”

“…Note however, that asking to reach a fixed target configuration like q F , 0 is a very different problem (cf. [2]). …”

“…We study the decidability of probability-1 qualitative reachability and Büchi objectives for infinite-state MDPs that are induced by suitable probabilistic extensions of VASS that we call VASS-MDPs. (Most quantitative objectives in probabilistic VASS are either undecidable, or the solution is at least not effectively expressible in (R, +, * , ≤) [2].) It is easy to show that, for general VASS-MDPs, even the simplest of these problems, (almost) sure reachability, is undecidable.…”

“…Techniques we used here are quite similar than the one presented in [2] to prove decidability of the sure reachability problems in probabilistic VASS (without nondeterminism).…”

Stochastic Parity Games on Lossy Channel Systems

“…Note that even if the players are restricted to finite memory strategies, such a strategy (even a memoryless one) on an infinite game graph is still an infinite 1 In the game community (e.g., [Zie98]) the word attractor is used to denote what we call a force set in Section 3. In the infinite-state systems community (e.g., [ABRS05,AHM07]), the word is used in the same way as we use it in this paper.…”