$$L^*$$-Based Learning of Markov Decision Processes

Abstract

Automata learning techniques automatically generate system models from test observations. These techniques usually fall into two categories: passive and active. Passive learning uses a predetermined data set, e.g., system logs. In contrast, active learning actively queries the system under learning, which is considered more efficient. An influential active learning technique is Angluin's L * algorithm for regular languages which inspired several generalisations from DFAs to other automata-based modelling formalisms. In this work, we study L *based learning of deterministic Markov decision processes, first assuming an ideal setting with perfect information. Then, we relax this assumption and present a novel learning algorithm that collects information by sampling system traces via testing. Experiments with the implementation of our sampling-based algorithm suggest that it achieves better accuracy than state-of-the-art passive learning techniques with the same amount of test data. Unlike existing learning algorithms with predefined states, our algorithm learns the complete model structure including the states.

L∗-based learning of Markov decision processes (extended version)

“…To the best of our knowledge, L * mdp is the first L*-based algorithm for MDPs that can be implemented via testing. Experimental results and the implementation can be found in the evaluation material for L * mdp [Tap20].…”

“…All tables include mean values along with the corresponding standard deviations separated by ±. Experimental results, the examined models, and the implementation of L * mdp can be found in the evaluation material [Tap20].…”

Learning Weighted Automata over Principal Ideal Domains

“…One easily adapts the example from Section 5 to show that learning probabilistic automata has a similar termination issue. On the positive side, Tappler et al [26] have shown that deterministic MDPs can be learned using an L based algorithm. The deterministic MDPs in loc.cit.…”

Runtime Monitors for Markov Decision Processes

“…Predictive monitoring has been combined with deep learning [17] and Bayesian inference [22], where the key problem is that the computation of an imminent failure is too expensive to be done exactly. More generally, learning automata models has been motivated with runtime assurance [1,55]. Testing approaches statistically evaluate whether traces are likely to be produced by a given model [25].…”