Neurochemical and Behavioral Dissections of Decision-Making in a Rodent Multistage Task

Abstract

Flexible decision-making in dynamic environments requires both retrospective appraisal of reinforced actions and prospective reasoning about the consequences of actions. These complementary reinforcement-learning systems can be characterized computationally with model-free and model-based algorithms, but how these processes interact at a neurobehavioral level in normal and pathological states is unknown. Here, we developed a translationally analogous multistage decision-making (MSDM) task to independently quantify modelfree and model-based behavioral mechanisms in rats. We provide the first direct evidence that male rats, similar to humans, use both model-free and model-based learning when making value-based choices in the MSDM task and provide novel analytic approaches for independently quantifying these reinforcement-learning strategies. Furthermore, we report that ex vivo dopamine tone in the ventral striatum and orbitofrontal cortex correlate with model-based, but not model-free, strategies, indicating that the biological mechanisms mediating decision-making in the multistage task are conserved in rats and humans. This new multistage task provides a unique behavioral platform for conducting systems-level analyses of decision-making in normal and pathological states. Significance StatementDecision-making is influenced by both a retrospective "model-free" system and a prospective "model-based" system in humans, but the biobehavioral mechanisms mediating these learning systems in normal and disease states are unknown. Here, we describe a translationally analogous multistage decision-making task to provide a behavioral platform for conducting neuroscience studies of decision-making in rats. We provide the first evidence that choice behavior in rats is influenced by model-free and model-based systems and demonstrate that model-based, but not model-free, learning is associated with corticostriatal dopamine tone. This novel behavioral paradigm has the potential to yield critical insights into the mechanisms mediating decision-making alterations in mental disorders.

Hierarchical Action Control: Adaptive Collaboration Between Actions and Habits

“…A number of reports have now been published evaluating two-stage discrimination learning in rodents (Akam et al, 2015;Miller et al, 2017;Groman et al, 2019). In a recent study, using a task modeled on that described for use in humans above, we sought to investigate how the state-space and action representations adapt to the structure of the world during the course of learning without any explicit instructions about the structure of the task -which obviously cannot be provided to rats (Dezfouli and Balleine, 2019) -see Figure 2E.…”

Individual differences in dopamine function underlying the balance between model-based and model-free control

“…A recent F-DOPA PET-fMRI study showed that individuals with higher ventral striatal DA synthesis capacity, indicating higher baseline DA [13], exert more reliance on MB than MF control, show stronger MB learning signals in the lateral prefrontal cortex (PFC) and weaker ventral striatal (VST) MF learning signals [8]. Using a rodent version of the same task, a positive correlation between baseline VST DA and MB control was recently confirmed [14]. Taken together, whereas the relationship to MF control remains unclear, a positive relationship between higher baseline DA and more MB control is supported by a series of human studies and one recent animal study.…”

The two‐step task, avoidance, and OCD

“…Developing rodent versions of the two‐step task is necessary for dissecting the neural circuits of model‐based and model‐free control. Currently four different versions of the task (Akam et al, ; Groman, Massi, Mathias, Curry, et al, ; Hasz & Redish, ; Miller, Botvinick, & Brody, ) have been described that differ from the human versions of the task in several important ways. Two of these tasks—one in mice (Akam et al, ) and the second in rats (Miller et al, ; Miller, Botnivick et al, )—do not have a second step choice.…”

“…A first step choice leads to a second step state, signaled via an illuminated nose poke, which leads to a probabilistic reward. However, the third and fourth tasks, in rats, maintain a second step choice (Groman, Massi, Mathias, Curry, et al, ; Hasz & Redish, ). The second main difference between rodent and human tasks involves how reward probabilities vary between trials.…”