Reward expectation selectively boosts the firing of accumbens D1+ neurons during motivated approach

Faust, Thomas W.; Mohebi, Ali; Berke, Joshua D.

doi:10.1101/2023.09.02.556060

Cited by 4 publications

(6 citation statements)

References 112 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At the level of the striatum, REDRL helps unify previous approaches to understanding D1 and D2 MSNs within a single, normative framework. While there have been previous hints that D1 and D2 MSNs are oppositely modulated by dopamine 22,39,40 and oppositely correlated with reward and expected value 5,6,[100][101][102] , this has generally been attributed to go/no-go or approach/avoid pathways and modeled using a single value predictor 3,66,113,120,121 . Here, we show how, far from being a bug or redundancy in the RL architecture, such diversity could actually be a feature, biasing convergence to optimistic or pessimistic value predictors.…”

Section: Discussionmentioning

confidence: 99%

“…We observed different patterns of deconvolved Ca 2+ activity across D1 and D2 populations [100][101][102] (Extended Data Fig. 10a-b).…”

Section: Opponent Contributions Of D1 and D2 Msns To Redrlmentioning

confidence: 93%

“…The opponency of the plasticity rule gives rise to opponent directions of value coding (Fig. 3f), with D1 5,6,100,101 and D2 MSNs 5,6,102 primarily correlating positively and negatively, respectively, with value and reward. Meanwhile, its asymmetric nature has the effect of extremizing value predictors — D1 MSNs are more optimistic, and D2 MSNs more pessimistic, than their individual dopamine inputs would create on their own (Fig.…”

Section: Using Striatal Opponency To Implement Distributional Rlmentioning

confidence: 99%

See 2 more Smart Citations

An opponent striatal circuit for distributional reinforcement learning

Lowet,

Zheng,

Meng

et al. 2024

Preprint

View full text Add to dashboard Cite

Machine learning research has achieved large performance gains on a wide range of tasks by expanding the learning target from mean rewards to entire probability distributions of rewards — an approach known as distributional reinforcement learning (RL)1. The mesolimbic dopamine system is thought to underlie RL in the mammalian brain by updating a representation of mean value in the striatum2,3, but little is known about whether, where, and how neurons in this circuit encode information about higher-order moments of reward distributions4. To fill this gap, we used high-density probes (Neuropixels) to acutely record striatal activity from well-trained, water-restricted mice performing a classical conditioning task in which reward mean, reward variance, and stimulus identity were independently manipulated. In contrast to traditional RL accounts, we found robust evidence for abstract encoding of variance in the striatum. Remarkably, chronic ablation of dopamine inputs disorganized these distributional representations in the striatum without interfering with mean value coding. Two-photon calcium imaging and optogenetics revealed that the two major classes of striatal medium spiny neurons — D1 and D2 MSNs — contributed to this code by preferentially encoding the right and left tails of the reward distribution, respectively. We synthesize these findings into a new model of the striatum and mesolimbic dopamine that harnesses the opponency between D1 and D2 MSNs5–15to reap the computational benefits of distributional RL.

show abstract

Section: Discussionmentioning

confidence: 99%

“…We observed different patterns of deconvolved Ca 2+ activity across D1 and D2 populations [100][101][102] (Extended Data Fig. 10a-b).…”

Section: Opponent Contributions Of D1 and D2 Msns To Redrlmentioning

confidence: 93%

Section: Using Striatal Opponency To Implement Distributional Rlmentioning

confidence: 99%

See 1 more Smart Citation

An opponent striatal circuit for distributional reinforcement learning

Lowet,

Zheng,

Meng

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…For all events we analyzed a 3 s window in 10 ms overlapping (5 ms) time bins. To determine significant modulation in all PETHs we used shuffle tests (Mohebi et al, 2019; Faust et al, 2023). Briefly we compared the firing rate of the neuron’s PETH in each time point to a shuffled distribution of 10,000 samples selected randomly from the same session.…”

Section: Methodsmentioning

confidence: 99%

A mismatch between striatal cholinergic pauses and dopaminergic reward prediction errors

Duhne,

Mohebi,

Kim

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Movement, motivation and reward-related learning depend strongly on striatal dopamine and acetylcholine. These neuromodulators each regulate the other, and disturbances to their coordinated signals contribute to human disorders ranging from Parkinson's Disease to depression and addiction. Pauses in the firing of cholinergic interneurons (CINs) are thought to coincide with pulses in dopamine release that encode reward prediction errors (RPEs), together shaping synaptic plasticity and thereby learning. However, such models are based upon recordings from unidentified neurons, and do not incorporate the distinct characteristics of striatal subregions. Here we compare the firing of identified, individual CINs to dopamine release as unrestrained rats performed a probabilistic decision-making task. The relationships between CIN spiking, dopamine release, and behavior varied strongly by subregion. In dorsal-lateral striatum a Go! cue evoked burst-pause CIN spiking, quickly followed by a very brief (~150ms) dopamine pulse that was unrelated to RPE. In dorsal-medial striatum the same cue evoked only a CIN pause; this pause was curtailed by a movement-selective rebound in firing. Finally in ventral striatum a reward cue evoked slower, RPE-coding increases in both dopamine and CIN firing, without any distinct pause. Our results demonstrate a spatial and temporal dissociation between CIN pauses and dopamine RPE signals, and will inform new models of striatal microcircuits and their contributions to behavior.

show abstract

“…Instead, they must learn this on their own beliefs by building an internal representation of the reward probabilities of each lever. Thus, this task is perhaps more challenging than one in which the rat is given external cues, 28,29 as it requires them to be more engaged in reinforcement learning and more vigilant in decision-making.…”

Section: Introductionmentioning

confidence: 99%

The flow of reward information through neuronal ensembles in the accumbens

Arroyo,

Hernandez-Lemus,

Gutierrez

2024

Preprint

View full text Add to dashboard Cite

The flow of reward information through neuronal ensembles in the nucleus accumbens shell (NAcSh) and its influence on decision-making remain poorly understood. We investigated these questions by training rats in a self-guided probabilistic choice task while recording single-unit activity in the NAcSh. We found that rats dynamically adapted their choices based on an internal representation of reward likelihood. NAcSh neurons encoded multiple task variables, including choices, outcomes (reward/no reward), and licking behavior. These neurons also exhibited sequential activity patterns resembling waves that peaked and dissipated with outcome delivery, potentially reflecting a global brain wave passing through the NAcSh. Further analysis revealed distinct neuronal ensembles processing distinct aspects of reward-guided behavior, further organized into four functionally specialized meta-ensembles. A Markov random fields graphical model revealed that NAcSh neurons form a small-world network with a heavy-tailed distribution, where most neurons have few functional connections and rare hubs are highly connected. This network architecture allows for efficient and robust information transmission. Neuronal ensembles exhibited dynamic interactions that reorganize depending on reward outcomes. Reinforcement learning within the session led to neuronal ensemble merging and increased network synchronization during reward delivery compared to omission. These findings offer a novel perspective of the flow of pleasure throughout neuronal ensembles in the NAcSh that dynamically changes its composition, with neurons dropping in and out, as the rat learns to obtain (energy) rewards in a changing environment and supports the idea that NAcSh ensembles encode the outcome of actions to guide decision making.

show abstract

Reward expectation selectively boosts the firing of accumbens D1+ neurons during motivated approach

Cited by 4 publications

References 112 publications

An opponent striatal circuit for distributional reinforcement learning

An opponent striatal circuit for distributional reinforcement learning

A mismatch between striatal cholinergic pauses and dopaminergic reward prediction errors

The flow of reward information through neuronal ensembles in the accumbens

Contact Info

Product

Resources

About