L∗-based learning of Markov decision processes (extended version)

Abstract

Automata learning techniques automatically generate systemmodels fromtest observations. Typically, these techniques fall into two categories: passive and active. On the one hand, passive learning assumes no interaction with the system under learning and uses a predetermined training set, e.g., system logs. On the other hand, active learning techniques collect training data by actively querying the system under learning, allowing one to steer the discovery ofmeaningful information about the systemunder learning leading to effective learning strategies. A notable example of active learning technique for regular languages is Angluin’s $$L^*$$ L ∗ -algorithm. The $$L^*$$ L ∗ -algorithm describes the strategy of a student who learns the minimal deterministic finite automaton of an unknown regular language $$L$$ L by asking a succinct number of queries to a teacher who knows $$L$$ L .In this work, we study $$L^*$$ L ∗ -based learning of deterministic Markov decision processes, a class of Markov decision processes where an observation following an action uniquely determines a successor state. For this purpose, we first assume an ideal setting with a teacher who provides perfect information to the student. Then, we relax this assumption and present a novel learning algorithm that collects information by sampling execution traces of the system via testing.Experiments performed on an implementation of our sampling-based algorithm suggest that our method achieves better accuracy than state-of-the-art passive learning techniques using the same amount of test obser vations. In contrast to existing learning algorithms which assume a predefined number of states, our algorithm learns the complete model structure including the state space.

Inferring Probabilistic Reward Machines from Non-Markovian Reward Signals for Reinforcement Learning

“…The following result follows from the convergence of L for label-deterministic MDPs, shown in (Tappler et al 2021). In that work, the authors show that under uniformly randomized testing strategies, the sampling-based L algorithm converges almost surely in the limit to the MDP under learning.…”

“…The L algorithm for learning regular languages (Angluin 1987) is the quintessential example of active inference, and assumes the existence of a minimally adequate teacher capable of answering membership and equivalence queries. This method has been broadly adopted and generalized to learn interface automata (Aarts and Vaan-drager 2010), Mealy machines (Niese 2003), automaton representations of recurrent neural networks Yahav 2018, 2019), and MDPs (Tappler et al 2021).…”

“…Lemma 1 ( (Tappler et al 2021)). Two MDPs M and M with n and at most n states, respectively, are equivalent iff…”

“…By resetting the learning rate parameter to very low values at the start of each query and gradually increasing it after a sufficiently long initial period of exploration results in period of essentially random exploration and thus simulates a uniformly random testing strategy. This allows us to apply the convergence arguments from (Tappler et al 2021) to our setting to assert that our learning procedure converges almost surely in the limit to the PRM (or an equivalent PRM) encoding the reward function of the TMDP under learning. In particular, lemma 1 provides a lower bound on the reinforcement learning episode lengths necessary to achieve convergence in the limit.…”

“…The reward-determinism property (c.f. definition 3) is inspired by the label-determinism property used for learning MDPs in (Tappler et al 2021). Algorithm 1 always learns a reward-deterministic PRM, even if the true reward is specified as a non-reward-deterministic PRM.…”