Learning optimal decisions for stochastic hybrid systems

Abstract

We apply reinforcement learning to approximate the optimal probability that a stochastic hybrid system satisfies a temporal logic formula. We consider systems with (non)linear continuous dynamics, random events following general continuous probability distributions, and discrete nondeterministic choices. We present a discretized view of states to the learner, but simulate the continuous system. Once we have learned a near-optimal scheduler resolving the choices, we use statistical model checking to estimate its probability of satisfying the formula. We implemented the approach using Q-learning in the tools HYPEG and modes, which support Petri net-and hybrid automata-based models, respectively. Via two case studies, we show the feasibility of the approach, and compare its performance and effectiveness to existing analytical techniques for a linear model. We find that our new approach quickly finds near-optimal prophetic as well as non-prophetic schedulers, which maximize or minimize the probability that a specific signal temporal logic property is satisfied.

The Modest State of Learning, Sampling, and Verifying Strategies

“…SMC also effectively works for non-Markovian and hybrid formalisms, as evidenced by modes' support for stochastic hybrid automata, however LSS does not [30]. We have recently combined SMC for such models with RL, but used explicitly stored Q-functions and discretisation for learning [86]. PMC approaches for such models are the subject of active research, with simple approaches based on interval abstractions provided by the Modest Toolset today [53,54].…”

“…Modest Tools. RL with an explicit representation of the Q-function is implemented in the Modest Toolset's modes tool to find strategies in non-linear stochastic hybrid automata, where classic PMC techniques cannot be applied due to the continuous nature of the state space [86].…”

ARCH-COMP23 Category Report: Stochastic Models

“…Continuous behavior that can be expressed by systems of ordinary differential equations can be simulated using an approximative approach [20,18], whereas piecewise-linear continuous behavior is simulated without approximation. HYPEG resolves discrete nondeterminism either probabilistically or using reinforcement learning to maximize or minimize the probability of a property [17,19], also in combination with a contract-based approach [2]. The tool is available at https://zivgitlab.uni-muenster.de/ag-sks/tools/HYPEG.…”