The Modest State of Learning, Sampling, and Verifying Strategies

Abstract

Optimal decision-making under stochastic uncertainty is a core problem tackled in artificial intelligence/machine learning (AI), planning, and verification. Planning and AI methods aim to find good or optimal strategies to maximise rewards or the probability of reaching a goal. Verification approaches focus on calculating the probability or reward, obtaining the strategy as a side effect. In this paper, we connect three strands of work on obtaining strategies implemented in the context of the Modest Toolset: statistical model checking with either lightweight scheduler sampling or deep learning, and probabilistic model checking. We compare their different goals and abilities, and show newly extended experiments on Racetrack benchmarks that highlight the tradeoffs between the methods. We conclude with an outlook on improving the existing approaches and on generalisations to continuous models, and emphasise the need for further tool development to integrate methods that find, evaluate, compare, and explain strategies.

Formal Methods Meet Machine Learning (F3ML)

“…In the previous section, we already discussed the paper "The Modest State of Learning, Sampling, and Verifying Strategies" [18] which discusses, among other topics, the applicability of statistical model checking to the task of verifying neural network based reinforcement learning systems. Thus, it interprets statistical model checking as a formal method that is applied to machine learning systems.…”

“…The differences, similarities and synergies between these approaches are highlighted in the paper "The Modest State of Learning, Sampling, and Verifying Strategies" [18] by Hartmanns, and Klauck. The paper connects probabilistic model checking and statistical model checking with scheduler sampling, as well as traditional Q-learning with value iteration methods and deep Q-learning.…”