Inferring Probabilistic Reward Machines from Non-Markovian Reward Signals for Reinforcement Learning

Abstract

The success of reinforcement learning in typical settings is predicated on Markovian assumptions on the reward signal by which an agent learns optimal policies. In recent years, the use of reward machines has relaxed this assumption by enabling a structured representation of non-Markovian rewards. In particular, such representations can be used to augment the state space of the underlying decision process, thereby facilitating non-Markovian reinforcement learning. However, these reward machines cannot capture the semantics of stochastic reward signals. In this paper, we make progress on this front by introducing probabilistic reward machines (PRMs) as a representation of non-Markovian stochastic rewards. We present an algorithm to learn PRMs from the underlying decision process and prove results around its correctness and convergence.

Learning Quadruped Locomotion Policies Using Logical Rules

“…Since the introduction of Reward Machines (RMs) (Icarte et al 2018), there have been various new research directions such as learning the RM structure (Xu et al 2020), RM for partially observable environments (Toro Icarte et al 2019), multi-agent intention scheduling (Dann et al 2022), and probabilistic RMs (Dohmen et al 2022) to name a few. While these works primarily focused on RM algorithmic im-provements and theoretical analysis, their applications did not go beyond toy domains.…”

“…In this paper, we alleviate the above mentioned problem of gait specification by leveraging Reward Machines (RMs) (Icarte et al 2022), which specify reward functions through deterministic finite automatons. RMs have been applied to various domains for guiding RL agents (Xu et al 2020;Neary et al 2020;Camacho et al 2021;Dohmen et al 2022). In this paper, RM serves as high-level specifications of gaits for low-level locomotion policy learning.…”

Learning Task Automata for Reinforcement Learning Using Hidden Markov Models

“…In contrast, our maximumlikelihood approach does not a priori require any structure of the specification or the spatial MDP environment. Meanwhile, [11,13,27,31,37] use Angluin [5]'s L * algorithm to learn a TA, relying on an oracle for equivalence and membership queries. We assume that the agent cannot access an oracle and must learn the TA fully autonomously, which aligns with the standard setup of model-free RL (note that L * was not originally developed for RL applications).…”

“…The most appropriate method will depend on the use-case as different approaches choose to relax different assumptions. For example, Corazza et al [9] and Dohmen et al [11] extend the SAT-based approach to noisy rewards and the L * approach to learning probabilistic automata, respectively.…”

“…Since our pipeline works in unknown environments where many high-level tasks may exist, agents using our pipeline can learn multiple tasks in one TA. Although we focus on deterministic TAs in this paper (i.e., DFAs), our approach can be straightforwardly extended to learning probabilistic automata [11,24]. This paper is self-contained, but see [2] for proposition proofs and experimental details.…”