SPOTTER: Extending Symbolic Planning Operators through Targeted Reinforcement Learning

Abstract

Symbolic planning models allow decision-making agents to sequence actions in arbitrary ways to achieve a variety of goals in dynamic domains. However, they are typically handcrafted and tend to require precise formulations that are not robust to human error. Reinforcement learning (RL) approaches do not require such models, and instead learn domain dynamics by exploring the environment and collecting rewards. However, RL approaches tend to require millions of episodes of experience and often learn policies that are not easily transferable to other tasks. In this paper, we address one aspect of the open problem of integrating these approaches: how can decision-making agents resolve discrepancies in their symbolic planning models while attempting to accomplish goals? We propose an integrated framework named SPOTTER that uses RL to augment and support ("spot") a planning agent by discovering new operators needed by the agent to accomplish goals that are initially unreachable for the agent. SPOTTER outperforms pure-RL approaches while also discovering transferable symbolic knowledge and does not require supervision, successful plan traces or any a priori knowledge about the missing planning operator.

Learning Neuro-Symbolic Relational Transition Models for Bilevel Planning

“…For example, previous work has considered learning propositional [Zhang et al, 2018, Dittadi et al, 2020, Tsividis, 2019 or lifted [Arora et al, 2018, Asai and Fukunaga, 2018, Asai, 2019, Asai and Muise, 2020, Ames et al, 2018, Ahmetoglu et al, 2020 symbolic transition models, and using them with AI planners [Hoffmann, 2001, Helmert, 2006. Other related work has used symbolic planners as managers in hierarchical RL, where low-level option policies are learned [Lyu et al, 2019, Sarathy et al, 2020, Gordon et al, 2019, Illanes et al, 2020, Yang et al, 2018, Kokel et al, 2021. In contrast to all these, we are focused on robotic settings where the planner must handle geometric considerations in addition to the symbolic ones.…”

Creativity of AI: Automatic Symbolic Option Discovery for Facilitating Deep Reinforcement Learning

“…The process of collecting experiences for learning is timeconsuming and the sample efficiency is low. To alleviate those issues, researchers have investigated the combination of HRL and symbolic planning to improve transferability, interpretability, and data efficiency (Ryan 2002;Leonetti, Iocchi, and Stone 2016;Yang et al 2018;Lyu et al 2019;Illanes et al 2020;Sarathy et al 2020;Lee et al 2021). In those works, the original MDP is divided into two levels.…”

“…A more realistic idea is to automatically learn action models from training data (Zhuo and Kambhampati 2017;Yang, Wu, and Jiang 2007;Ng and Petrick 2019;Martínez et al 2016;James, Rosman, and Konidaris 2020) and exploit the learnt action models to generate plans for guiding the exploration of options. Although there is indeed an approach (Sarathy et al 2020) proposed to learn action models automatically, they still need to manually define major parts of action models in advance. Besides, the planning goal in this approach is kept unchanged while in our framework it is dynamically adapted to maximize the external reward.…”

Creativity of AI: Automatic Symbolic Option Discovery for Facilitating Deep Reinforcement Learning

“…To alleviate those issues, researchers have investigated the combination of h-DRL and symbolic planning to improve transferability, interpretability, and data efficiency (Parr and Russell 1997;Ryan 2002;Hogg, Kuter, and Muñoz-Avila 2010;Leonetti, Iocchi, and Stone 2016;Yang et al 2018;Lyu et al 2019;Illanes et al 2020;Sarathy et al 2020). In this structure, the original MDP is divided into two levels.…”

“…A more realistic idea is to automatically learn action models from training data (Zhuo and Kambhampati 2017;Yang, Wu, and Jiang 2007;Ng and Petrick 2019;Martínez et al 2016;James, Rosman, and Konidaris 2020) and exploit the learnt action models to generate plans for guiding the exploration of options. Although there is indeed an approach (Sarathy et al 2020) proposed to learn action models automatically, they still need to manually define a major part of the models in advance. Besides, the planning goal in this approach is kept unchanged while it is dynamically adapted to maximize the external reward in our framework.…”