Statistical Model Checking for Markov Decision Processes

Henriques, David; Martins, João Carlos; Zuliani, Paolo; Platzer, André; Clarke, Edmund M.

doi:10.1109/qest.2012.19

Cited by 85 publications

(106 citation statements)

References 24 publications

Supporting

Mentioning

104

Contrasting

Order By: Relevance

“…In contrast, we show that the statistical approach from [20] does not always converge (see the example in the next section).…”

Section: Introductionmentioning

confidence: 78%

“…Both the learning-based and the baseline approximate algorithms significantly improve upon a state-of-the-art statistical model checking algorithm, originally developed for MDPs [20]. That algorithm also uses sampling and reinforcement learning, but it needs to sample multiple (possibly many) times along the same path to obtain a good estimate of the quality function used for reinforcement [37].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Exact and approximate probabilistic symbolic execution for nondeterministic programs

Luckow

Păsăreanu

Dwyer

et al. 2014

Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering

View full text Add to dashboard Cite

Probabilistic software analysis seeks to quantify the likelihood of reaching a target event under uncertain environments. Recent approaches compute probabilities of execution paths using symbolic execution, but do not support nondeterminism. Nondeterminism arises naturally when no suitable probabilistic model can capture a program behavior, e.g., for multithreading or distributed systems.In this work, we propose a technique, based on symbolic execution, to synthesize schedulers that resolve nondeterminism to maximize the probability of reaching a target event. To scale to large systems, we also introduce approximate algorithms to search for good schedulers, speeding up established random sampling and reinforcement learning results through the quantification of path probabilities based on symbolic execution.We implemented the techniques in Symbolic PathFinder and evaluated them on nondeterministic Java programs. We show that our algorithms significantly improve upon a stateof-the-art statistical model checking algorithm, originally developed for Markov Decision Processes.

show abstract

“…In contrast, we show that the statistical approach from [20] does not always converge (see the example in the next section).…”

Section: Introductionmentioning

confidence: 78%

Section: Introductionmentioning

confidence: 99%

Exact and approximate probabilistic symbolic execution for nondeterministic programs

Luckow

Păsăreanu

Dwyer

et al. 2014

Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering

View full text Add to dashboard Cite

show abstract

“…For the former, some SMC-like approaches have recently been developed. They either work by iteratively optimising the decisions of an explicitly-stored scheduler [4,9], or by sampling from the scheduler space and iteratively improving a set of candidate near-optimal schedulers [5]. The former are heavyweight techniques because the size of the description of the (memoryless) scheduler is significant, and in the worst case is the size of the state space.…”

Section: Introductionmentioning

confidence: 99%

Statistical Approximation of Optimal Schedulers for Probabilistic Timed Automata

D’Argenio

Hartmanns

Legay

et al. 2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. The verification of probabilistic timed automata involves finding schedulers that optimise their nondeterministic choices with respect to the probability of a property. In practice, approaches based on model checking fail due to state-space explosion, while simulation-based techniques like statistical model checking are not applicable due to the nondeterminism. We present a new lightweight on-the-fly algorithm to find near-optimal schedulers for probabilistic timed automata. We make use of the classical region and zone abstractions from timed automata model checking, coupled with a recently developed smart sampling technique for statistical verification of Markov decision processes. Our algorithm provides estimates for both maximum and minimum probabilities. We compare our new approach with alternative techniques, first using tractable examples from the literature, then motivate its scalability using case studies that are intractable to numerical model checking and challenging for existing statistical techniques.

show abstract

“…Existing results in temporal logic-constrained verification and control synthesis with unknown systems are mainly in two categories: The first uses statistical model checking and hypothesis testing for Markov chains [6] and MDPs [7]. The second applies inference algorithms to identify the unknown factors and adapt the controller with the inferred model (a probabilistic automaton, or a two-player deterministic game) of the system and its environment [8,9].…”

Section: Introductionmentioning

confidence: 99%

“…The second applies inference algorithms to identify the unknown factors and adapt the controller with the inferred model (a probabilistic automaton, or a two-player deterministic game) of the system and its environment [8,9]. Statistical model checking for MDPs [7] relies on sampling of the trajectories of Markov chains induced from the underlying MDP and policies to verify whether the probability of satisfying a bounded linear temporal logic constraint is greater than some quantity for all admissible policies. It is restricted to bounded linear temporal logic properties in order to make the sampling and checking for paths computationally feasible.…”

Section: Introductionmentioning

confidence: 99%

Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints

Topcu²

2014

Robotics: Science and Systems X

113

102

View full text Add to dashboard Cite

Abstract-We consider synthesis of controllers that maximize the probability of satisfying given temporal logic specifications in unknown, stochastic environments. We model the interaction between the system and its environment as a Markov decision process (MDP) with initially unknown transition probabilities. The solution we develop builds on the so-called model-based probably approximately correct Markov decision process (PAC-MDP) method. The algorithm attains an ε-approximately optimal policy with probability 1−δ using samples (i.e. observations), time and space that grow polynomially with the size of the MDP, the size of the automaton expressing the temporal logic specification,and a finite time horizon. In this approach, the system maintains a model of the initially unknown MDP, and constructs a product MDP based on its learned model and the specification automaton that expresses the temporal logic constraints. During execution, the policy is iteratively updated using observation of the transitions taken by the system. The iteration terminates in finitely many execution steps. With high probability, the resulting policy is such that, for any state, the difference between the probability of satisfying the specification under this policy and the optimal one is within a predefined bound.

show abstract

Statistical Model Checking for Markov Decision Processes

Cited by 85 publications

References 24 publications

Exact and approximate probabilistic symbolic execution for nondeterministic programs

Exact and approximate probabilistic symbolic execution for nondeterministic programs

Statistical Approximation of Optimal Schedulers for Probabilistic Timed Automata

Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints

Contact Info

Product

Resources

About