Deep reinforcement learning (RL) methods have driven impressive advances in artificial intelligence in recent years, exceeding human performance in domains ranging from Atari to Go to no-limit poker. This progress has drawn the attention of cognitive scientists interested in understanding human learning. However, the concern has been raised that deep RL may be too sample-inefficientthat is, it may simply be too slowto provide a plausible model of how humans learn. In the present review, we counter this critique by describing recently developed techniques that allow deep RL to operate more nimbly, solving problems much more quickly than previous methods. Although these techniques were developed in an AI context, we propose that they may have rich implications for psychology and neuroscience. A key insight, arising from these AI methods, concerns the fundamental connection between fast RL and slower, more incremental forms of learning. Powerful but Slow: The First Wave of Deep RL Over just the past few years, revolutionary advances have occurred in artificial intelligence (AI) research, where a resurgence in neural network or 'deep learning' methods [1,2] has fueled breakthroughs in image understanding [3,4], natural language processing [5,6], and many other areas. These developments have attracted growing interest from psychologists, psycholinguists, and neuroscientists, curious about whether developments in AI might suggest new hypotheses concerning human cognition and brain function [7-11]. One area of AI research that appears particularly inviting from this perspective is deep RL (Box 1). Deep RL marries neural network modeling (see Glossary) with reinforcement learning, a set of methods for learning from rewards and punishments rather than from more explicit instruction [12]. After decades as an aspirational rather than practical idea, deep RL has within the past 5 years exploded into one of the most intense areas of AI research, generating superhuman performance in tasks from video games [13] to poker [14], multiplayer contests [15], and complex board games, including go and chess [16-19]. Highlights Recent AI research has given rise to powerful techniques for deep reinforcement learning. In their combination of representation learning with rewarddriven behavior, deep reinforcement learning would appear to have inherent interest for psychology and neuroscience.

Alchemy is a new meta-learning environment rich enough to contain interesting abstractions, yet simple enough to make fine-grained analysis tractable. Further, Alchemy provides an optional symbolic interface that enables meta-RL research without a large compute budget. In this work, we take the first steps toward using Symbolic Alchemy to identify design choices that enable deep-RL agents to learn various types of abstraction. Then, using a variety of behavioral and introspective analyses we investigate how our trained agents use and represent abstract task variables, and find intriguing connections to the neuroscience of abstraction. We conclude by discussing the next steps for using meta-RL and Alchemy to better understand the representation of abstract variables in the brain.Preprint. Under review.