Computational heterogeneity in the human mesencephalic dopamine system

Abstract

Recent evidence in animals has indicated that the mesencephalic dopamine system is heterogeneous anatomically, molecularly, and functionally, and it has been suggested that the dopamine system comprises distinct functional systems. Identifying and characterizing these systems in humans will have widespread ramifications for understanding drug addiction and mental health disorders. Model-based studies in humans have suggested an analogous computational heterogeneity, in which dopaminergic targets in striatum encode both experience-based learning signals and counterfactual learning signals that are based on hypothetical information. We used brainstem-tailored fMRI to identify mesencephalic sources of experiential and counterfactual learning signals. Participants completed a decision-making task based on investing in markets. This sequential investment task generated experience-based learning signals, in the form of temporal difference (TD) reward prediction errors, and counterfactual learning signals, in the form of “fictive errors.” Fictive errors are reinforcement learning signals based on hypothetical information about “what could have been.” An additional learning signal was constructed to be relatable to a motivational salience signal. Blood oxygenation level dependent responses in regions of substantia nigra (SN) and ventral tegmental area (VTA), where dopamine neurons are located, coded for TD and fictive errors, and additionally were related to the motivational salience signal. These results are highly consistent with animal electrophysiology and provide direct evidence that human SN and VTA heterogeneously handle important reward-harvesting computations.Electronic supplementary materialThe online version of this article (doi:10.3758/s13415-013-0191-5) contains supplementary material, which is available to authorized users.

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

“…Here we demonstrate a segregation between effort and reward learning within SN/VTA across the domains of task activation, functional connectivity, and gray matter density. In SN/VTA, a dorsomedial encoding of reward PEs, and a ventrolateral encoding of effort PEs, extends previous studies on SN/VTA subregions (56,57,67,68) by demonstrating that this segregation has functional implications that are exploited during multiattribute learning. In contrast to previous studies on SN/VTA substructures (56, 67-69), we performed whole-brain imaging, which allowed us to investigate the precise interactions between dopaminergic midbrain and striatal/ cortical areas.…”

“…In contrast to previous studies on SN/VTA substructures (56, 67-69), we performed whole-brain imaging, which allowed us to investigate the precise interactions between dopaminergic midbrain and striatal/ cortical areas. However, this required a slightly lower spatial SN/ VTA resolution than previous studies (56,(67)(68)(69), restricting our analyses to spatial gradients across the entire SN/VTA rather than subregion analyses. We speculate that the dorsomedial region showing reward PE activity is likely to correspond to a dorsal tier of dopamine neurons known to form mesolimbic connections projecting to VS regions (55) (SI Appendix, Fig.…”

“…Although dopaminergic activity cannot be assessed directly using fMRI, both effort and reward PEs were evident in segregated regions of dopaminergic-rich midbrain, and where an effective connectivity analysis indicated a directional influence from SN/VTA toward subcortical (reward PE) and cortical (effort PE) targets via ascending mesolimbic and mesocortical pathways, respectively. Dopaminergic midbrain is thought to comprise several distinct dopaminergic populations that have dissociable functions (54,56,67,68). Here we demonstrate a segregation between effort and reward learning within SN/VTA across the domains of task activation, functional connectivity, and gray matter density.…”

Dopamine Does Double Duty in Motivating Cognitive Effort

“…A normative account appeals to the opportunity costs of working memory allocation (Kurzban et al, 2013). Evidence of DA encoding opportunity costs comes from a high-resolution FMRI study finding signed RPE-like increases in activity in the VTA/SN corresponding with the value of unchosen options which therefore constituted missed opportunities (D'Ardenne et al, 2013). The ACC, by virtue of its connectivity with lateral PFC working memory circuits, e.g.…”

Computational neuroimaging strategies for single patient predictions

“…Model-based fMRI has subsequently been applied to numerous domains of cognition, accommodating a diversity of modelling approaches and computational themes, such as reinforcement learning (RL) models of behaviour and Bayesian models of cognition (e.g., (D'Ardenne et al, 2013;Daw et al, 2006;Iglesias et al, 2013;Klein-Flügge et al, 2011;Schwartenbeck et al, 2015;Seymour et al, 2004;Vossel et al, 2015). A recent application of model-based fMRI has been the investigation of interactions between learning and decision-making processes which do or do not derive from an explicit model of the environment or task structure.…”