Prefrontal Cortex as a Meta-Reinforcement Learning System

Abstract

1Over the past twenty years, neuroscience research on reward-based learning has converged on a canonical model, under which the neurotransmitter dopamine 'stamps in' associations between situations, actions and rewards by modulating the strength of synaptic connections between neurons. However, a growing number of recent findings have placed this standard model under strain. In the present work, we draw on recent advances in artificial intelligence to introduce a new theory of rewardbased learning. Here, the dopamine system trains another part of the brain, the prefrontal cortex, to operate as its own free-standing learning system. This new perspective accommodates the findings that motivated the standard model, but also deals gracefully with a wider range of observations, providing a fresh foundation for future research.Exhilarating advances have recently been made toward understanding the mechanisms involved in reward-driven learning. This progress has been enabled in part by the importation of ideas from the field of reinforcement learning 1 (RL). Most centrally, this input has led to an RL-based theory of dopaminergic function. Here, phasic dopamine (DA) release is interpreted as conveying a reward prediction error (RPE) signal [2][3][4] , an index of surprise which figures centrally in temporal-difference RL algorithms 1 . Under the theory, the RPE drives synaptic plasticity in the striatum, translating experienced action-reward associations into optimized behavioral policies 4,5 . Over the past two decades, evidence has steadily mounted for this proposal, establishing it as the standard model of reward-driven learning.However, even as this standard model has solidified, a collection of problematic observations has accumulated. One quandary arises from research on prefrontal cortex (PFC). A growing body of evidence suggests that PFC implements mechanisms for reward-based learning, performing computations that strikingly resemble those ascribed to DA-based RL. It has long been established that sectors of the PFC represent the expected values of actions, objects and states [6][7][8] . More recently, it has emerged that PFC also encodes the recent history of actions and rewards [9][10][11][12][13][14][15] . The set of variables encoded, along with observations concerning the temporal profile of neural activation in the PFC, has led to the conclusion that "PFC neurons dynamically [encode] conversions from reward and choice history to object value, and from object value to object choice" 10 . In short, neural activity in PFC appears to reflect a set of operations that together constitute a self-contained RL algorithm.Placing PFC beside DA, we obtain a picture containing two full-fledged RL systems, one utilizing activity-based representations and the other synaptic learning. What is the relationship between these systems? If both support RL, are their functions simply redundant? One suggestion has been that DA and PFC subserve different forms of learning, with DA implementing model-free RL, based on direct stim...

Multi-Task Reinforcement Learning in Humans

“…Beyond the algorithms examined in our paper, there are a number of other important ideas that have been successful in machine learning. For example, hierarchical RL algorithms learn policy primitives that combine to produce solutions to different tasks, 11 while meta-RL algorithms learn a learning algorithm that can adapt quickly to new tasks, 12,13 echoing the classic formation of task sets described by Harlow. 14 Of particular note is the development of hybrids of UVFAs and SF&GPI known as universal successor features approximators, which combine the benefits of both approaches.…”

“…Yang and colleagues 15 trained a single recurrent neural network to solve multiple related tasks and observed the emergence of functionally specialized clusters of neurons, mixed selectivity neurons like those found in macaque prefrontal cortex, as well as compositional task representations reminiscent of hierarchical RL. Wang and colleagues 12 proposed how meta-RL might be implemented in the brain, with dopamine signals gradually training a separate learning algorithm in the prefrontal cortex, which in turn can rapidly adapt to changing task demands.…”

Models of heterogeneous dopamine signaling in an insect learning and memory center

“…Another recent study used a meta-learning approach to model DA activity and activity in the prefrontal cortex (PFC) of mammals (Wang et al, 2018). Unlike our study, in which the "slow" optimization is taken to represent evolutionary and developmental processes that determine the MB output circuitry, in this study the slow component of learning involved DA-dependent optimization of recurrent connections in PFC.…”

Computational neural mechanisms of goal-directed planning and problem solving

“…Both MBRL and MFRL suffer from this limitation. For example, Q-learning (Watkins & Dayan, 1992) (Wang et al, 2018). MBRL fares little better.…”