Rare Events for Statistical Model Checking an Overview

Legay, Axel; Sedwards, Sean; Traonouez, Louis-Marie

doi:10.1007/978-3-319-45994-3_2

Cited by 11 publications

(7 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We see this article as a first step in bringing practices from the statistical model checking (SMC) tool-set to the ABM computational economics community. Of particular interest in this respect are SMC techniques developed to mitigate two classic problems of Monte Carlo methods: dealing with models that present rare events (Legay et al, 2016), and using machine learning techniques to reduce the number of simulations (Bortolussi et al, 2015). Finally, we will expand the family of automated analysis techniques offered in MultiVeStA.…”

Section: Discussionmentioning

confidence: 99%

Automated and Distributed Statistical Analysis of Economic Agent-Based Models

Vandin¹,

Giachini²,

Lamperti³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a novel approach to the statistical analysis of simulation models and, especially, agent-based models (ABMs). Our main goal is to provide a fully automated and model-independent tool-kit to inspect simulations and perform counter-factual analysis. Our approach: (i) is easy-to-use by the modeller, (ii) improves reproducibility of results, (iii) optimizes running time given the modeller's machine, (iv) automatically chooses the number of required simulations and simulation steps to reach user-specified statistical confidence, and (v) automatically performs a variety of statistical tests. In particular, our framework is designed to distinguish the transient dynamics of the model from its steady state behaviour (if any), estimate properties of the model in both "phases", and provide indications on the ergodic (or non-ergodic) nature of the simulated processes -which, in turns allows one to gauge the reliability of a steady state analysis. Estimates are equipped with statistical guarantees, allowing for robust comparisons across computational experiments. To demonstrate the effectiveness of our approach, we apply it to two models from the literature: a large scale macro-financial ABM and a small scale prediction market model. Compared to prior analyses of these models, we obtain new insights and we are able to identify and fix some erroneous conclusions.

show abstract

Section: Discussionmentioning

confidence: 99%

Automated and Distributed Statistical Analysis of Economic Agent-Based Models

Vandin¹,

Giachini²,

Lamperti³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Some cases remain problematic, such as systems where states are visited very rarely. Nevertheless, we foresee potential solutions involving rare event simulation [ 21 ]. This goes beyond the scope of this work and it is left to future work.…”

Section: Discussionmentioning

confidence: 99%

Global PAC Bounds for Learning Discrete Time Markov Chains

Bazille

Genest

Jégourel

et al. 2020

Computer Aided Verification

View full text Add to dashboard Cite

Learning models from observations of a system is a powerful tool with many applications. In this paper, we consider learning Discrete Time Markov Chains (DTMC), with different methods such as frequency estimation or Laplace smoothing . While models learnt with such methods converge asymptotically towards the exact system, a more practical question in the realm of trusted machine learning is how accurate a model learnt with a limited time budget is. Existing approaches provide bounds on how close the model is to the original system, in terms of bounds on local (transition) probabilities, which has unclear implication on the global behavior. In this work, we provide global bounds on the error made by such a learning process, in terms of global behaviors formalized using temporal logic . More precisely, we propose a learning process ensuring a bound on the error in the probabilities of these properties. While such learning process cannot exist for the full LTL logic, we provide one ensuring a bound that is uniform over all the formulas of CTL. Further, given one time-to-failure property, we provide an improved learning algorithm. Interestingly, frequency estimation is sufficient for the latter, while Laplace smoothing is needed to ensure non-trivial uniform bounds for the full CTL logic.

show abstract

“…It provides several SMC algorithms that can be applied to different types of systems and properties using a plugin system. Plasma Lab's algorithms include estimation algorithm with Monte Carlo method, hypothesis testing with SPRT, estimation of rare events with importance splitting and importance sampling [20], and algorithms for Markov decision processes [10]. Plasma Lab includes a simulator for the Reactive Module Language (RML) of the probabilistic modelchecker Prism [17] that allows to specify discrete and continuous time Markov chains, as well as Markov decision processes.…”

Section: Implementation Using Plasma Labmentioning

confidence: 99%

Statistical Model Checking of Incomplete Stochastic Systems

Arora

Legay

Richmond

et al. 2018

Leveraging Applications of Formal Methods, Verification and Validation. Verification

Self Cite

View full text Add to dashboard Cite

We study incomplete stochastic systems that are missing some parts of their design, or are lacking information about some components. It is interesting to get early analysis results of the requirements of these systems, in order to adequately refine their design. In previous works, models for incomplete systems are analysed using model checking techniques for three-valued temporal logics. In this paper, we propose statistical model checking algorithms for these logics. We illustrate our approach on a case-study of a network system that is refined after the analysis of early designs.

show abstract

Rare Events for Statistical Model Checking an Overview

Abstract: Abstract. This invited paper surveys several simulation-based approaches to compute the probability of rare bugs in complex systems. The paper also describes how those techniques can be implemented in the professional toolset Plasma.

Cited by 11 publications

References 22 publications

Automated and Distributed Statistical Analysis of Economic Agent-Based Models

Automated and Distributed Statistical Analysis of Economic Agent-Based Models

Global PAC Bounds for Learning Discrete Time Markov Chains

Statistical Model Checking of Incomplete Stochastic Systems

Contact Info

Product

Resources

About