Integration of Reinforcement Learning and Optimal Decision-Making Theories of the Basal Ganglia

Abstract

This article seeks to integrate two sets of theories describing action selection in the basal ganglia: reinforcement learning theories describing learning which actions to select to maximize reward and decision-making theories proposing that the basal ganglia selects actions on the basis of sensory evidence accumulated in the cortex. In particular, we present a model that integrates the actor-critic model of reinforcement learning and a model assuming that the cortico-basal-ganglia circuit implements a statistically optimal decision-making procedure. The values of cortico-striatal weights required for optimal decision making in our model differ from those provided by standard reinforcement learning models. Nevertheless, we show that an actor-critic model converges to the weights required for optimal decision making when biologically realistic limits on synaptic weights are introduced. We also describe the model's predictions concerning reaction times and neural responses during learning, and we discuss directions required for further integration of reinforcement learning and optimal decision-making theories.

Dopamine reward prediction error signal codes the temporal evaluation of a perceptual decision report

“…It has been suggested that when the brain does not have full access to the correct value of the physical attributes of the stimuli, the cerebral cortex uses noisy observations to infer them (39,40) and that the midbrain DA neurons and striatal neuronal circuits evaluate the state of the environment to select the appropriate actions based on the results of that inference (35)(36)(37). In this scheme, the outcome of the inference process is a posterior probability about the state of the environment, which is interpreted as a measure of the belief about that state (41).…”

“…This is the basic scheme followed in early proposals about how to extend the RL framework to model the DA activity in decision-making tasks (35)(36)(37). In this approach, the belief state is used to predict rewards, to compute the error in the prediction, and to select the action that indicates the final choice.…”

“…To investigate these issues further we have taken a different approach, proposing a model based on the RL framework and using it to interpret the activity of DA neurons. Because of the uncertainty on the stimulus amplitude and on trial duration, the model estimates the total reward and RPEs using belief states (35)(36)(37).…”

Errors in action timing and inhibition facilitate learning by tuning distinct mechanisms in the underlying decision process

“…Despite the underlying intuition that intelligent behavior requires both the ability to control actions as well as to learn from consequent feedback, it is not well understood how these learning and control systems interact. One reason for this is that computational models of learning [1,2] and control [3][4][5] have historically emerged from disparate lines of empirical research (see [6,7] for exceptions), adding difficulty to the already challenging task of inferring cognitive phenomena from gross behavioral measures. Recently, however, insights from cognitive and computational neuroscience have begun to shed light on the interaction of cognitive processes in neural circuits, providing additional empirical anchors for grounding theoretical assumptions [8][9][10][11] .…”

“…The basal ganglia (BG), a subcortical network that shares dense, reciprocal connections with much of cortex, as well as many subcortical neuromodulators, is known to play a critical role in both learning and control processes -acting as a central integration hub where cortically-distributed control commands [12][13][14] , sensory evidence [15,16] , and decision variables [17][18][19] can be weighed and synthesized with feedback-dependent learning signals to facilitate goal-directed behavior [7,20,21] . Cortical information enters the BG through three primary pathways: the first two of these pathways, entering via the striatum, are the direct (i.e., facilitating; Fig 1A, green) and indirect pathways (i.e., suppressing; Fig 1A, blue).…”