Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas

Abstract

We demonstrate a reinforcement learning agent which uses a compositional recurrent neural network that takes as input an LTL formula and determines satisfying actions. The input LTL formulas have never been seen before, yet the network performs zero-shot generalization to satisfy them. This is a novel form of multi-task learning for RL agents where agents learn from one diverse set of tasks and generalize to a new set of diverse tasks. The formulation of the network enables this capacity to generalize. We demonstrate this ability in two domains. In a symbolic domain, the agent finds a sequence of letters in that are accepted. In a Minecraft-like environment, the agent finds a sequence of actions that conform to the formula. While prior work could learn to execute one formula reliably given examples of that formula, we demonstrate how to encode all formulas reliably. This could form the basis of new multitask agents that discover sub-tasks and execute them without any additional training, as well as the agents which follow more complex linguistic commands. The structures required for this generalization are specific to LTL formulas, which opens up an interesting theoretical question: what structures are required in neural networks for zero-shot generalization to different logics?

Compositional Reinforcement Learning from Logical Specifications

“…Camacho et al [8] show that one can generate RMs from temporal specifications but RMs generated this way lead to sparse rewards. Kuo et al [23] propose a compositional model for zero-shot execution of LTL formulas but training such a model requires a lot of samples even for relatively simpler environments. There has also been recent work on using temporal logic specifications for multi-agent RL [13,29].…”

In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications

“…Formally, the first component of the SM is the extractor E, which transforms the complex formula T into a list K consisting of all the sequences of atomic tasks α that satisfy T . As it is common in the literature (Kuo et al, 2020;Vaezipoor et al, 2021), we assume that the SM have access to an internal labelling function L I : Z → 2 AP . L I is the restriction of L to the observation space and maps the observations of the agent in Z into the set AP of atoms.…”

“…Beyond TL, we find methods such as reward machines (a type of finite state machine) Xu et al, 2020;, or RL-specific formal languages such as SPECTRL (Jothimurugan et al, 2019). Closer to our line of work, Kuo et al (2020) presents a novel RL framework to follow OOD combinations of known tasks expressed in linear-time temporal logic (LTL) by training multiple networks (one per LTL operator and per object). In a similar line, Araki et al (2021) introduces a hierarchical reinforcement learning framework aimed to learn policies that are optimal and composable while relying on different neural networks each specialized in one subtask.…”

“…Formal languages (Huth & Ryan, 2004) offer desirable properties such as unambiguous semantics and compositional syntax, allowing to automatically generate large amounts of training instructions and their corresponding reward functions. Earlier contributions in this area rely on the compositional nature of formal languages, often employing multiple policy networks to execute temporal logic (TL) instructions, e.g., one network per sub-task within the given instruction (Andreas et al, 2017; or one per symbol and object (Kuo et al, 2020). However, these approaches do not scale well with the number of instructions since policy networks are a computationally costly resource and, consequently, these earlier studies are restricted to relatively small state-spaces environments that require less computation, e.g., non-visual settings.…”

Compositional RL Agents That Follow Language Commands in Temporal Logic

“…This manuscript is an extension of “Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas” ( Kuo et al, 2020b ) by the same authors published at the Conference on Intelligent Robots and Systems (IROS) 2020. The manuscript contains over 30% new material, including additional technical details and a new domain, Fetch, which required substantive advances.…”