Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network

Pan, Wei-Xing; Schmidt, Robert; Wickens, Jeffery R.; Hyland, Brian I.

doi:10.1523/jneurosci.1478-05.2005

Cited by 406 publications

(462 citation statements)

References 44 publications

Supporting

Mentioning

412

Contrasting

Order By: Relevance

“…Consistent with previous reports 2, [5][6][7][8][9][10][11][12][13] , this population included many neurons (n = 19) that increased firing in response to reward. Notably, activity in the 19 dopamine neurons that responded to either cue or reward (cue/reward-responsive dopamine neurons) reflected the positive and negative reward prediction errors that are inherent in our task design, whereas activity in the remaining 14 cue-responsive neurons did not.…”

Section: Prediction Error Signaling In Cue-responsive Neuronssupporting

confidence: 85%

See 1 more Smart Citation

Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards

2007

View full text Add to dashboard Cite

The dopamine system is thought to be involved in making decisions about reward. Here we recorded from the ventral tegmental area in rats learning to choose between differently delayed and sized rewards. As expected, the activity of many putative dopamine neurons reflected reward prediction errors, changing when the value of the reward increased or decreased unexpectedly. During learning, neural responses to reward in these neurons waned and responses to cues that predicted reward emerged. Notably, this cue-evoked activity varied with size and delay. Moreover, when rats were given a choice between two differently valued outcomes, the activity of the neurons initially reflected the more valuable option, even when it was not subsequently selected.Dopamine is thought to be essential for reinforcement learning 1-4 . This hypothesis is rooted in single-unit studies, which show that the firing patterns of dopamine neurons accord well with a signal that reports errors in reward prediction 5-15 . This is true of firing for rewards as well as for cues that come to predict rewards. This signal is thought to regulate learning and to guide decision-making; however, few studies have investigated this issue in behavioral tasks that involve active decision-making 11,14 .To address this issue, we recorded single-unit activity in the ventral tegmental area (VTA) of rats performing a variant of a time-discounting task. It is known that animals prefer immediate over delayed reward (time discounting) 16-25 , but the role of dopamine in this impulsive behavior remains unclear 26-30 . We designed a task to isolate changes in neural activity related to reward size from changes in neural activity related to time to reward by independently varying when (after a short or long delay) or what (big or small) reward was to be delivered at the end of the trial. As the rats learned to respond to the more valuable odor cue, many dopamine neurons developed cue-selective activity during odor sampling, firing more strongly for cues that predicted either the more immediate reward or the larger one. Notably, these same neurons exhibited activity that was consistent with prediction errors during unexpected delivery or omission of reward. When the rats were given the opportunity to choose between two

show abstract

Section: Prediction Error Signaling In Cue-responsive Neuronssupporting

confidence: 85%

“…Waveforms from these neurons (Fig. 2b) were analyzed for features characteristic of dopamine neurons [12][13][14]32,33 (Fig. 2c).…”

Section: Resultsmentioning

confidence: 99%

Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards

2007

View full text Add to dashboard Cite

show abstract

“…However, as noted earlier, simple TD models (but not PVLV) also predict a chaining of DA bursts "backward in time" from the reward to the stimulus onset, which has not been reliably observed empirically (Fiorillo, Tobler, & Schultz, 2005;Pan et al, 2005). However, this particular aspect of the data is still controversial (e.g., Niv, Duff, & Dayan, 2005) and also depends critically on the way that the input environment is represented.…”

Section: Application To Conditioning Datamentioning

confidence: 90%

“…For example, Pan et al (2005) recently showed how a TD( ) model with a high lambda value could reproduce the empirically observed pattern (i.e., no evidence of backward marching dopamine bursts). Furthermore, the data often show dopamine bursts at both the CS and US (Pan et al, 2005;Schultz, 2002)-this is incompatible with 1 A simpler possible implementation would be to just add the two delta values to produce a summed DA value, but this double counts the rewardrelated deltas because both the LV and PV contribute in this case. Nevertheless, because LV and PV deltas otherwise occur at different times, Equation 11 is very similar to adding the deltas; the PV system just dominates when external rewards are presented or expected.…”

Section: Application To Conditioning Datamentioning

confidence: 99%

PVLV: The Primary Value and Learned Value Pavlovian Learning Algorithm.

O’Reilly

Frank

Hazy

et al. 2007

Behavioral Neuroscience

112

115

View full text Add to dashboard Cite

The authors present their primary value learned value (PVLV) model for understanding the rewardpredictive firing properties of dopamine (DA) neurons as an alternative to the temporal-differences (TD) algorithm. PVLV is more directly related to underlying biology and is also more robust to variability in the environment. The primary value (PV) system controls performance and learning during primary rewards, whereas the learned value (LV) system learns about conditioned stimuli. The PV system is essentially the Rescorla-Wagner/delta-rule and comprises the neurons in the ventral striatum/nucleus accumbens that inhibit DA cells. The LV system comprises the neurons in the central nucleus of the amygdala that excite DA cells. The authors show that the PVLV model can account for critical aspects of the DA firing data, making a number of clear predictions about lesion effects, several of which are consistent with existing data. For example, first-and second-order conditioning can be anatomically dissociated, which is consistent with PVLV and not TD. Overall, the model provides a biologically plausible framework for understanding the neural basis of reward learning.Keywords: basal ganglia, dopamine, reinforcement learning, Pavlovian conditioning, computational modeling An important and longstanding challenge for both the cognitive neuroscience and artificial intelligence communities has been to develop an adequate understanding (and a correspondingly robust model) of Pavlovian learning. Such a model should account for the full range of signature findings in the rich literature on this phenomenon. Pavlovian conditioning refers to the ability of previously neutral stimuli that reliably co-occur with primary rewards to elicit new conditioned behaviors and to take on reward value themselves (e.g., Pavlov's famous case of the bell signaling food for hungry dogs; Pavlov, 1927).Pavlovian conditioning is distinguished from instrumental conditioning in that the latter involves the learning of new behaviors that are reliably associated with reward, either first order (US), or second order (CS). Although Pavlovian conditioning also involves behaviors (conditioned and unconditioned responses), reward delivery is not contingent on behavior but is instead reliably paired with a stimulus regardless of behavior. In contrast, instrumental conditioning explicitly makes reward contingent on a particular "operant" or "instrumental" response. Both stimulus-reward (Pavlovian) and stimulus-response-reward (instrumental) associations, however, are thought to be trained by the same phasic dopamine signal that occurs at the time of primary reward (US) as described below. In practice, the distinction is often blurry as the two types of conditioning interact (e.g., second-order instrumental conditioning and so-called Pavlovian instrumental transfer effects).The dominant theoretical perspective for both Pavlovian and instrumental conditioning since the seminal Rescorla and Wagner (1972) model, is that learning is based on the discrepancy between act...

show abstract

“…As illustrated in Fig. 2, during classical conditioning, unexpected reward initially triggered an increase in phasic activity of DA neurons, which then shifted with learning to the conditioned stimulus (Ljungberg et al 1992;Mirenowicz and Schultz 1994;Pan et al 2005). When an expected reward was omitted, the activity of DA neurons paused at the precise time when the reward should have been delivered (Ljungberg et al 1992;Roesch et al 2007;Satoh et al 2003).…”

Section: Introductionmentioning

confidence: 94%

Computational models of reinforcement learning: the role of dopamine as a reward signal

2010

View full text Add to dashboard Cite

Reinforcement learning is ubiquitous. Unlike other forms of learning, it involves the processing of fast yet content-poor feedback information to correct assumptions about the nature of a task or of a set of stimuli. This feedback information is often delivered as generic rewards or punishments, and has little to do with the stimulus features to be learned. How can such low-content feedback lead to such an efficient learning paradigm? Through a review of existing neuro-computational models of reinforcement learning, we suggest that the efficiency of this type of learning resides in the dynamic and synergistic cooperation of brain systems that use different levels of computations. The implementation of reward signals at the synaptic, cellular, network and system levels give the organism the necessary robustness, adaptability and processing speed required for evolutionary and behavioral success.

show abstract

Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network

Cited by 406 publications

References 44 publications

Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards

Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards

PVLV: The Primary Value and Learned Value Pavlovian Learning Algorithm.

Computational models of reinforcement learning: the role of dopamine as a reward signal

Contact Info

Product

Resources

About