Symbolic Plans as High-Level Instructions for Reinforcement Learning

Abstract

Reinforcement learning (RL) agents seek to maximize the cumulative reward obtained when interacting with their environment. Users define tasks or goals for RL agents by designing specialized reward functions such that maximization aligns with task satisfaction. This work explores the use of high-level symbolic action models as a framework for defining final-state goal tasks and automatically producing their corresponding reward functions. We also show how automated planning can be used to synthesize high-level plans that can guide hierarchical RL (HRL) techniques towards efficiently learning adequate policies. We provide a formal characterization of taskable RL environments and describe sufficient conditions that guarantee we can satisfy various notions of optimality (e.g., minimize total cost, maximize probability of reaching the goal). In addition, we do an empirical evaluation that shows that our approach converges to near-optimal solutions faster than standard RL and HRL methods and that it provides an effective framework for transferring learned skills across multiple tasks in a given environment.

RePReL: Integrating Relational Planning and Reinforcement Learning for Effective Abstraction

“…Finally, the use of relational representation for planning allows appropriate generalization including the number and types of objects in the domain without excessive feature engineering. Our results in 4 compelling domains show that RePReL significantly outperforms the state-of-the-art Planner+RL combination (Illanes et al 2020) while achieving better generalization and transfer.…”

“…Several prior works have explored the idea of combining a planner and RL agents to solve complex problems which have some notion of temporally extended actions or task hierarchies (Grounds and Kudenko 2005;Yang et al 2018;Lyu et al 2019;Jiang et al 2019;Eppe, Nguyen, and Wermter 2019). Among these, RePReL is closely related to the Taskable RL framework of Illanes et al (2020). Similar to Taskable RL , RePReL employs a planner to generate useful instructions (task definitions) for the RL agent.…”

“…Any opinions, findings, conclusion or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of the ARO, AFOSR, NSF, DARPA or the US government. We sincerely thank Illanes et al (2020) for sharing the Taskable RL code for baselines.…”

A survey of knowledge‐based sequential decision‐making under uncertainty

“…In some RDK-for-SDM systems, RDK and prior experiences of executing actions in the domain are used to construct domain models and guide SDM. For instance, symbolic planning has been combined with hierarchical RL to guide the agent's interactions with the world, resulting in reliable world models and SDM (Illanes et al 2020). In other work, each symbolic transition is mapped (manually) to options, that is, temporally extended MDP actions; RDK helps compute the MDP models and policies, and the outcomes of executing the corresponding primitive actions help revise the values of state action combinations in the symbolic reasoner (Yang et al 2018).…”

Guiding Robot Exploration in Reinforcement Learning via Automated Planning

“…Hierarchical RL and Automated Planning: Planning methods have been used to guide the higher level of hierarchical RL methods (Icarte et al 2018;Yang et al 2018;Lyu et al 2019;Jiang et al 2019a;Illanes et al 2020;Gordon, Fox, and Farhadi 2019). In those methods, the agents use an action language to compute plans to decompose a complex task into a sequence of subtasks, and each subtask is then implemented by a reinforcement learner.…”