Continual Reinforcement Learning with Multi-Timescale Successor Features

Abstract

In Deep Reinforcement Learning (RL), it is a challenge to learn representations that do not exhibit catastrophic forgetting or interference in non-stationary environments. Successor Features (SFs) offer a potential solution to this challenge. However, canonical techniques for learning SFs from pixel-level observations often lead to representation collapse, wherein representations degenerate and fail to capture meaningful variations in the data. More recent methods for learning SFs can avoid representation collapse, but they often involve complex losses and multiple learning phases, reducing their efficiency. We introduce a novel, simple method for learning SFs directly from pixels. Our approach uses a combination of a Temporaldifference (TD) loss and a reward prediction loss, which together capture the basic mathematical definition of SFs. We show that our approach matches or outperforms existing SF learning techniques in both 2D (Minigrid), 3D (Miniworld) mazes and Mujoco, for both single and continual learning scenarios. As well, our technique is efficient, and can reach higher levels of performance in less time than other approaches. Our work provides a new, streamlined technique for learning SFs directly from pixel observations, with no pretraining required 1 .

The successor representation subserves hierarchical abstraction for goal-directed behavior

“…The successor representation (SR) (18) is a reinforcement learning algorithm specifically concerned with learning task structure based on a biologically plausible learning rule (19)(20)(21). Notably, it has been suggested that the successor representation can be leveraged to discover hierarchical representations (22,23). The successor representation learns a multi-step prediction of the task structure, which allows for fast and flexible behavior in the face of changing goals (24).…”