Large-Scale Retrieval for Reinforcement Learning

Abstract

Effective decision making involves flexibly relating past experiences and relevant contextual information to a novel situation. In deep reinforcement learning, the dominant paradigm is for an agent to amortise information that helps decisionmaking into its network weights via gradient descent on training losses. Here, we pursue an alternative approach in which agents can utilise large-scale contextsensitive database lookups to support their parametric computations. This allows agents to directly learn in an end-to-end manner to utilise relevant information to inform their outputs. In addition, new information can be attended to by the agent, without retraining, by simply augmenting the retrieval dataset. We study this approach in Go, a challenging game for which the vast combinatorial state space privileges generalisation over direct matching to past experiences. We leverage fast, approximate nearest neighbor techniques in order to retrieve relevant data from a set of tens of millions of expert demonstration states. Attending to this information provides a significant boost to prediction accuracy and game-play performance over simply using these demonstrations as training trajectories, providing a compelling demonstration of the value of large-scale retrieval in reinforcement learning agents.

Using Memory-Based Learning to Solve Tasks with State-Action Constraints

“…At test time, a relatively small subset of these experiences are indexed and a local model is fit to them. Memory based learning variants have been used for both robot control [2][3][4] [5], and reinforcement learning [6][7] [8]. Prior work has highlighted the advantages of memory-based learning including efficient learning, ease of adding new experiences to the agent by simply storing them in memory, the avoidance of catastrophic interference, and the effects of distribution shifts and long tailed distributions.…”