Abstraction-Refinement for Hierarchical Probabilistic Models

Abstract

Markov decision processes are a ubiquitous formalism for modelling systems with non-deterministic and probabilistic behavior. Verification of these models is subject to the famous state space explosion problem. We alleviate this problem by exploiting a hierarchical structure with repetitive parts. This structure not only occurs naturally in robotics, but also in probabilistic programs describing, e.g., network protocols. Such programs often repeatedly call a subroutine with similar behavior. In this paper, we focus on a local case, in which the subroutines have a limited effect on the overall system state. The key ideas to accelerate analysis of such programs are (1) to treat the behavior of the subroutine as uncertain and only remove this uncertainty by a detailed analysis if needed, and (2) to abstract similar subroutines into a parametric template, and then analyse this template. These two ideas are embedded into an abstraction-refinement loop that analyses hierarchical MDPs. A prototypical implementation shows the efficacy of the approach.

Accurately Computing Expected Visiting Times and Stationary Distributions in Markov Chains

“…Moreover, EVTs have been leveraged for minimizing and learning DTMCs [1,2]. Further recent applications of EVTs to MDPs include verifying cause-effect dependencies [3], as well as an abstraction-refinement procedure that measures the importance of states based on the EVTs under a fixed policy [33]. [22] employs EVTs in the context of policy iteration in reward-robust MDPs.…”

Pareto Curves for Compositionally Model Checking String Diagrams of MDPs

“…Related Work Compositional verification methods for sequential MDPs [6,21,26,33] have been discussed in §1. For hierarchical MCs [1], there are no schedulers.…”

“…This type of compositionality allows to reduce the peak memory consumption by reasoning about the individual parts and allows to exploit the typical existence of isomorphic parts of the state space. Sequentially composed MDPs have seen a surge in interest recently [20,21,26,32,33].…”

“…So far, this problem has been circumvented in the literature on compositional MDP verification. In [21], the notion of locally optimal policies is used. In essence, the technique relies on syntactic restrictions, such as open MDPs without nondeterminism or with a dedicated desirable exit.…”

Parameter Synthesis in Markov Models: A Gentle Survey

“…In particular, the methods described here can be extended towards parametric probabilistic timed automata [43] and to controller synthesis for uncertain POMDPs, see below. Similarly, there exist various approaches for parametric continuous-time MCs, see, e.g., [16,18,39] and parameter synthesis has been applied to stochastic population models [41] and to accelerate solving hierarchical MDPs [54].…”