Options of Interest: Temporal Abstraction with Interest Functions

Abstract

Temporal abstraction refers to the ability of an agent to use behaviours of controllers which act for a limited, variable amount of time. The options framework describes such behaviours as consisting of a subset of states in which they can initiate, an internal policy and a stochastic termination condition. However, much of the subsequent work on option discovery has ignored the initiation set, because of difficulty in learning it from data. We provide a generalization of initiation sets suitable for general function approximation, by defining an interest function associated with an option. We derive a gradient-based learning algorithm for interest functions, leading to a new interest-option-critic architecture. We investigate how interest functions can be leveraged to learn interpretable and reusable temporal abstractions. We demonstrate the efficacy of the proposed approach through quantitative and qualitative results, in both discrete and continuous environments.

Reusability and Transferability of Macro Actions for Reinforcement Learning

“…The concept of macro actions has been adopted in the domain of planning [Asai and Fukunaga 2015;Botea et al 2005;Chrpa and Vallati 2019;Coles and Smith 2007;DeJong and Mooney 1986;Kaelbling 1993;Khetarpal et al 2020;Korf 1985;Newton et al 2007;Sacerdoti 1974], and has been shown to be able to provide advantages such as the embedding effect and evaluation effect [Botea et al 2005]. The former enables bypassing a series of successor states from a start state, and thus allows the search space to be changed as well as the search depth to be reduced.…”

The Paradox of Choice: Using Attention in Hierarchical Reinforcement Learning

“…1(a)) places "soft" preferences over action choices, but does not rule out any of them. Interest functions (Khetarpal et al, 2020b) for options can be used to implement this idea. Appropriately tuned interest can reduce the noise in an agent's behavior.…”

“…While our implementation of interest over option choices (i.e. soft-attention) is closely related to the interest function introduced in interest-option-critic (IOC, Khetarpal et al, 2020b), there are two key differences: 1) soft attention in our approach is learned using downstream-task-agnostic intent completion targets, as opposed to a task-specific reward function in IOC, and 2) we use a softmax over predicted discounted intent completion values, instead of gradient-based updates to parameterized interest as in IOC. Alg.…”

“…A well-known solution is to select a small number of choices to focus on, for instance, applying an action only when certain preconditions are met in classical planning (Fikes et al, 1972), or using initiation sets for options (Sutton et al, 1999). Recent work (Khetarpal et al, 2020b) proposed a generalization of initiation sets to interest functions (Sutton et al, 2016;White, 2017), which provide a way to learn options that specialize to certain regions of the state space. All these approaches can be viewed as providing an attention mechanism over action choices.…”

Temporally Abstract Partial Models

“…Much of the work on learning and planning with options considers that they apply everywhere (Bacon et al, 2017;Harb et al, 2017;Harutyunyan et al, 2019b,a), with some notable recent exceptions which generalize the notion of initiation sets in the context of function approximation (Khetarpal et al, 2020b). Having options that are partially defined is very important in order to control the complexity of the planning and exploration process.…”