Dual Competition between the Basal Ganglia and the Cortex: from Action-Outcome to Stimulus-Response

Abstract

Action-outcome (A-O) and stimulus-response (S-R) processes that are two forms of instrumental conditioning that are important components of decision making and action selection. The former adapts its response according to the outcome while the latter is insensitive to the outcome. An unsolved question is how these two processes emerge, cooperate and interact inside the brain in order to issue a unique behavioral answer. Here we propose a model of the interaction between the cortex, the basal ganglia and the thalamus based on a dual competition. We hypothesize that the striatum, the subthalamic nucleus, the internal pallidum (GPi), the thalamus, and the cortex are involved in closed feedback loops through the hyperdirect and direct pathways. These loops support a competition process that results in the ability for the basal ganglia to make a cognitive decision followed by a motor decision. Considering lateral cortical interactions (short range excitation, long range inhibition), another competition takes place inside the cortex allowing this latter to make a cognitive and a motor decision. We show how this dual competition endows the model with two regimes. One is oriented towards action-outcome and is driven by reinforcement learning, the other is oriented towards stimulus-response and is driven by Hebbian learning. The final decision is made according to a combination of these two mechanisms with a gradual transfer from the former to the latter. We confirmed these theoretical results on primates using a two-armed bandit task and a reversible bilateral inactivation of the internal part of the globus pallidus.

Habits without values.

“…These models propose that novel behaviors are first acquired via a dopamine-dependent plasticity mechanism within the basal ganglia, and that with consistent performance, control of behavior is transferred to cortex via a Hebbian cortico–cortical plasticity mechanism. This idea has been applied to categorization learning and instrumental conditioning (Ashby et al, 2007), sequence learning (Hélie, Roeder, Vucovich, Rünger, & Ashby, 2015) and to action selection in probabilistic environments (Topalidou, Kase, Boraud, & Rougier, 2017), and it has been suggested to describe how basal-ganglia-dependent behavior becomes automatic in general (Hélie, Ell, & Ashby, 2015). Though the cortical module of these models is conceptually similar to our habitual controller, the basal ganglia module is different from our goal-directed controller in important ways: its learning rule instantiates a version of model-free RL, which tends to repeat actions in situations where they have led to reinforcement in the past, but does not learn about the particular outcomes that are expected to follow each action.…”

Habits without Values

“…These models propose that novel behaviors are first acquired via a dopamine-dependent plasticity mechanism within the basal ganglia, and that with consistent performance, control of behavior is transferred to cortex via a Hebbian cortico-cortical plasticity mechanism. Developed first in the context of categorization learning (Ashby, Ennis, & Spiering, 2007) , this idea has recently been applied to sequence learning (Hélie, Roeder, Vucovich, Rünger, & Ashby, 2015) and to action selection in probabilistic environments (Topalidou, Kase, Boraud, & Rougier, 2017) , and it has been suggested to describe how basal-ganglia-dependent behavior becomes automatic in general (Hélie, Ell, & Ashby, 2015) . Though the cortical module of these models is conceptually similar to our habitual controller, the basal ganglia module is different from our goal-directed controller in important ways: its learning rule instantiates a version of model-free RL, which tends to repeat actions in situations where they have led to reinforcement in the past, but does not learn about the particular outcomes that are expected to follow each action.…”