Distributional coding of associative learning within projection-defined populations of midbrain dopamine neurons

Abstract

Midbrain dopamine neurons are thought to play key roles in learning by conveying the difference between expected and actual outcomes. While this teaching signal is often considered to be uniform, recent evidence instead supports diversity in dopamine signaling. However, it remains poorly understood how heterogeneous signals might be organized to facilitate the role of downstream circuits mediating distinct aspects of behavior. Here we investigated the organizational logic of dopaminergic signaling by recording and labeling individual midbrain dopamine neurons during associative behavior. We defined combinations of protein expression and cellular localization to sort recorded neurons according to the striatal regions they innervate. Our findings show that reward information and task variables are not only heterogeneously encoded, with multiplexing, but also differentially distributed across populations of dopamine neurons projecting to different regions of striatum. These data, supported by computational modelling, indicate that such distributional coding can maximize dynamic range and tailor dopamine signals to facilitate the specialized roles of different striatal regions.

A feature-specific prediction error model explains dopaminergic heterogeneity

“…Formally investigating this prediction (e.g., by systematically comparing cue and outcome responding across different projection-defined DA populations) is the key open empirical test of our framework. However, existing data are generally consistent with the hypothesis that the sorts of heterogeneity associated with outcome-specific differences tend to arise between distinct DA nuclei or target regions 7,90,91 , in contrast to the hallmarks of feature-specific PEs we study here, which occur within VTA 4,20 and also within SNc 5 . Thus, in SNc and DMS, compared to VTA and NAc, movement responses tend to emerge while reward responses decline (a combination consistent with outcome-rather than feature-specific variation; Fig.…”

Vector-valued dopamine improves learning of continuous outputs in the striatum

“…A vector-valued error would likely appear as tunings to various motor and task variables in experimental animals (Requirement 2 ), especially in the phase before the animals are so overtrained their error is zero. Indeed, cells in the SNr (Fan et al, 2012;Barter et al, 2015;Tang et al, 2021) as well as the SNc (Howe and Dombeck, 2016;Dodson et al, 2016;Coddington and Dudman, 2018;Avvisati et al, 2022) respond to a plethora of behavioral and task variables.…”

“…A vector-valued error would likely appear as tunings to various motor and task variables in experimental animals ( Requirement 2 ), especially in the phase before the animals are so overtrained their error is zero. Indeed, cells in the SNr (Fan et al, 2012; Barter et al, 2015; Tang et al, 2021) as well as the SNc (Howe and Dombeck, 2016; Dodson et al, 2016; Coddington and Dudman, 2018; Avvisati et al, 2022) respond to a plethora of behavioral and task variables. We deliberately excluded the details of how this error may be computed in the brain, but we speculate at least three possible algorithmic ways in which it could appear: Bran region regions such as motor cortex or cerebellum could have a forward model of the world as well as the target, and thus, can directly compute the error and send it to the midbrain. The brain could be wired as a set of hierarchical control loops, in which each loop provides the target for the level below (as proposed by Yin, 2014).…”

“…In principle one could imagine this problem being solved by representing each action dimension in a spatially compact region that receives its private dopamine channel (Hamid et al (2021) and Lee et al (2022) have proposed similar ideas). Firstly however, experimental evidence suggests both individual striatal projection neurons (Rueda-Orozco and Robbe, 2015;Klaus et al, 2017;Weglage et al, 2021) and individual dopaminergic neurons (Avvisati et al, 2022) have mixed tuning rather than responding to a single task variable, as would be predicted from a naive parallel organization. Secondly, although there is a coarse-grain somatotopic organisation of the striatum (Hunnicutt et al, 2016) as well as the substantia nigra (Foster et al, 2021), the axonal arborizations of individual SNc neurons are huge and cover large portions of the striatum (up to 5%; Matsuda et al, 2009).…”

Dopamine neurons encode a multidimensional probabilistic map of future reward

“…Recently, TD algorithms have been elaborated to include a set of value learning channels that differ in their sensitivity to positive and negative RPEs, leading to value estimates that converge to distinct statistics of the expected cumulative reward distribution, an innovation termed distributional RL 6 (Figure 1B). Such innovations have been shown to improve performance of deep RL agents on benchmark tasks due to improved statistical robustness, and evidence of distributional RL-like computations has been reported in midbrain DANs of both mice and primates 7,8,9 , however the direct functional relevance of such distributional mechanisms and representations to behavior is unknown. In the engineering setting, deep RL agents vary in whether and how they make use of knowledge about the distribution over future rewards when selecting actions [10][11][12] , and decoding of reward distributions from DAN activity has only been demonstrated at the time of reward delivery 7 .…”