Automated Experiment Design for Data-Efficient Verification of Parametric Markov Decision Processes

Abstract

Abstract. We present a new method for statistical verification of quantitative properties over a partially unknown system with actions, utilising a parameterised model (in this work, a parametric Markov decision process) and data collected from experiments performed on the underlying system. We obtain the confidence that the underlying system satisfies a given property, and show that the method uses data efficiently and thus is robust to the amount of data available. These characteristics are achieved by firstly exploiting parameter synthesis to establish a feasible set of parameters for which the underlying system will satisfy the property; secondly, by actively synthesising experiments to increase amount of information in the collected data that is relevant to the property; and finally propagating this information over the model parameters, obtaining a confidence that reflects our belief whether or not the system parameters lie in the feasible set, thereby solving the verification problem.

Parameter-Independent Strategies for pMDPs via POMDPs

“…Consensus is the exception, and we have solved some instances of Consensus as reported below. In the examples used in [36], showing how to obtain policies that optimize learning the parameter values, the optimal policy is again independent of the parameters.…”

“…In the past years, there has been a great effort to solve reachability analysis in pMDPs using symbolic approaches [15,21,37,16,1,36]. These methods generally partition the parameter space in regions, associating each region to the optimal memoryless scheduler that maximizes/minimizes the probability to reach the target state.…”

Data-Driven Verification under Signal Temporal Logic Constraints

“…Satisfaction of properties expressed in linear temporal logic on finite traces for linear time-invariant (LTI) systems is investigated in Haesaert et al (2015Haesaert et al ( , 2016 by using Bayesian inference. The proposed approach in Polgreen et al (2017) applies Bayesian inference to parametric Markov chain.…”

ABC(SMC)$$^2$$: Simultaneous Inference and Model Checking of Chemical Reaction Networks

“…[22] computes the probability that an underlying stochastic system satisfies a given property using data produced by the system and leveraging system's models. Along this line of work, the integration of verification of parameterised discrete-time Markov chains and Bayesian inference is considered in [38], with an extension to Markov decision processes in [39]. Both [38,39] work with small finite-state models with fully observable traces, which allows the posterior probability distribution to be calculated analytically and parameters to be synthesised symbolically.…”

“…Along this line of work, the integration of verification of parameterised discrete-time Markov chains and Bayesian inference is considered in [38], with an extension to Markov decision processes in [39]. Both [38,39] work with small finite-state models with fully observable traces, which allows the posterior probability distribution to be calculated analytically and parameters to be synthesised symbolically. On the contrary, here we work with partially observed data and stochastic models with intractable likelihoods, and must rely on likelihood-free methods and statistical parameter synthesis procedures.…”