A neural network model for the orbitofrontal cortex and task space acquisition during reinforcement learning

Abstract

Reinforcement learning has been widely used in explaining animal behavior. In reinforcement learning, the agent learns the value of the states in the task, collectively constituting the task state space, and uses the knowledge to choose actions and acquire desired outcomes. It has been proposed that the orbitofrontal cortex (OFC) encodes the task state space during reinforcement learning. However, it is not well understood how the OFC acquires and stores task state information. Here, we propose a neural network model based on reservoir computing. Reservoir networks exhibit heterogeneous and dynamic activity patterns that are suitable to encode task states. The information can be extracted by a linear readout trained with reinforcement learning. We demonstrate how the network acquires and stores task structures. The network exhibits reinforcement learning behavior and its aspects resemble experimental findings of the OFC. Our study provides a theoretical explanation of how the OFC may contribute to reinforcement learning and a new approach to understanding the neural mechanism underlying reinforcement learning.

Economic Choices under Simultaneous or Sequential Offers Rely on the Same Neural Circuit

“…Similar results were obtained in monkeys choosing between juice bundles (Pastor-Bernier et al, 2019), in mice (Kuwabara et al, 2020), and in humans using fMRI (Hare et al, 2008;Howard et al, 2015). The variables encoded in OFC capture both the input and the output of the choice process, and the corresponding cell groups are computationally sufficient to generate binary decisions (Rustichini and Padoa-Schioppa, 2015;Song et al, 2017;Zhang et al, 2018). In monkeys, mild electrical stimulation of this area biases choices in predictable ways (Ballesta et al, 2020).…”

“…These variables capture both the input and the output of the choice process, suggesting that the cell groups identified in OFC might constitute the building blocks of e decision circuit. The population dynamics (Rich and Wallis, 2016), correlations between neuronal and behavioral variability (Padoa-Schioppa, 2013), the effects of lesion (Camille et al, 2011;Yu et al, 2018) or inactivation (Gore et al, 2019;Kuwabara et al, 2020), and computational modeling (Rustichini and Padoa-Schioppa, 2015;Song et al, 2017;Zhang et al, 2018) support this proposal. These and corroborating results set the stage for a detailed understanding of the decision mechanisms.…”

Models that learn how humans learn: The case of decision-making and its disorders

“…Recurrent neural networks have previously been used to study reward-related decision-making [13, 14], perceptual decision-making, performance in cognitive tasks, working-memory [15, 16, 17, 18, 19, 20], motor patterns, motor reach and timing [21, 22, 23, 24]. Typically, in these studies, an rnn is itself trained to perform the task.…”

“…Here to address these limitations we consider an alternative approach based on recurrent neural networks (RNNs); a flexible class of models that make minimal assumptions about the underlying learning processes used by the subject and that are known to have sufficient capacity to represent any form of computational process [12], including those believed to be behind the behaviour of humans and other animals in a wide range of decision-making, cognitive and motor tasks [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]. Since these models are flexible, they can automatically characterize the major behavioral trends exhibited by real subjects without requiring tweaking and engineering.…”

The role of executive function in shaping reinforcement learning

“…Simplifying the state space is a function sometimes attributed to attentional filters, which can specify important features of the environment [14]. In this framework, attention tags the features of the environment that RL variables are computed over [32,33,34,14]. This is accomplished by attention differentially weighing environmental features, assigning a higher weight to task-relevant ones [33] ( Figure 2).…”