A gradual temporal shift of dopamine responses mirrors the progression of temporal difference error in machine learning

Abstract

It has been proposed that the activity of dopamine neurons approximates temporal difference (TD) prediction error, a teaching signal developed in reinforcement learning, a field of machine learning. However, whether this similarity holds true during learning remains elusive. In particular, some TD learning models predict that the error signal gradually shifts backward in time from reward delivery to a reward-predictive cue, but previous experiments failed to observe such a gradual shift in dopamine activity. Here we demonstrate conditions in which such a shift can be detected experimentally. These shared dynamics of TD error and dopamine activity narrow the gap between machine learning theory and biological brains, tightening a long-sought link..

Pupillary Dynamics of Mice Performing a Pavlovian Delay Conditioning Task Reflect Reward-Predictive Signals

“…In the future, examining the pharmacological manipulation of dopamine D1 receptors is an important step for better circuit-level understanding of the neuronal mechanism of the reward prediction and the reward prediction error. Dopamine neurons in VTA show phasic activity to unexpected reward presentations, but phasic activity to the reward decreases as learning progresses, and the neurons show phasic activity to reward-predictive cues (Amo et al, 2022;Schultz et al, 1997). Thus, in our experiments, increases in pupil size may reflect reward prediction errors in the presentation of the auditory stimulus.…”

“…However, the neuronal activities of LC neurons in the study of Bouret and Sara (2003) were examined with the reversal of the contingency between the stimulus and the outcome or the re-acquisition after the extinction in the Go/No-Go task. Although the phasic activities to unpredicted reward found in LC neurons (Bouret and Sara, 2004) may be slightly different from those neurons found in dopaminergic neurons found in VTA (Amo et al, 2022;Schultz et al, 1997), both phasic activities of LC and DA neurons are known to show phasic responses to unpredictable events. Moreover, LC neurons show phasic activity in response to a novel stimulus and decreased activity when the stimulus ceases to predict biologically important events (Berridge and Waterhouse, 2003;Vankov et al, 1995).…”

“…LC neurons show a burst of activity when the stimuli that predict biologically important events, such as reward and aversive events, are presented (Aston-Jones ad Bloom, 1981;Aston-Jones et al, 1997, 2005Bouret and Richmond, 2015;Bouret and Sara, 2004). LC neurons also show similar activities to dopaminergic neurons in the ventral tegmental area (VTA), such as increased phasic activity in response to unpredicted reward and decreased activity through repeated experience and transfer to a reward-predicting stimulus (Amo et al, 2022;Bouret and Sara, 2004;Schultz et al, 1997). However, the neuronal activities of LC neurons in the study of Bouret and Sara (2003) were examined with the reversal of the contingency between the stimulus and the outcome or the re-acquisition after the extinction in the Go/No-Go task.…”

“…Dopamine neurons in VTA show phasic activity to unexpected reward presentations, but phasic activity to the reward decreases as learning progresses, and the neurons show phasic activity to reward-predictive cues (Amo et al, 2022; Schultz et al, 1997). Thus, in our experiments, increases in pupil size may reflect reward prediction errors in the presentation of the auditory stimulus.…”

Prefrontal norepinephrine represents a threat prediction error under uncertainty

“…The state model has a learningrelated temporal shift which leads to rapidly diminished cue-onset prediction errors in the 15 and 30s conditions at 20 trials of training (Figure 5C), in contrast to the data. This occurs due to a moving prediction error that moves backward in time from shock to cue onset over trials (Supplementary Figure 4A-B), a phenomenon which has been recently observed in dopamine neuron responses to appetitive learning (Amo et al, 2022). Such a prediction error can only "arrive" at the cue onset quickly if the distance between cue onset and shock is short.…”