Model-based predictions for dopamine

Langdon, Angela J.; Sharpe, Melissa J; Schoenbaum, Geoffrey; Niv, Yaron

doi:10.1016/j.conb.2017.10.006

Cited by 139 publications

(130 citation statements)

References 69 publications

Supporting

Mentioning

121

Contrasting

Order By: Relevance

“…These results are consistent with the proposal that the midbrain dopamine system signals a generalized prediction error, reflecting a failure to predict features of an unexpected event beyond and even orthogonal to value (Gardner et al, 2018;Howard and Kahnt, 2018;Langdon et al, 2017;Takahashi et al, 2017). Importantly this proposal is not necessarily contrary to current canon; it can account for value errors as a special example of a more general function (Gardner et al, 2018), one readily apparent in the firing of individual neurons perhaps due to the priority given to such information when it is the goal of the experimental subject.…”

Section: Discussionsupporting

confidence: 90%

“…However, the same neurons also respond to errors in predicting the features of rewarding events, even when their value remains unchanged (Howard and Kahnt, 2018;Takahashi et al, 2017). Such sensory prediction errors would be useful for learning detailed information about the relationships between real-world events (Gardner et al, 2018;Howard and Kahnt, 2018;Langdon et al, 2017;Takahashi et al, 2017). Indeed, dopamine transients facilitate learning such relationships, independent of value, when they are appropriately positioned to mimic endogenous errors Keiflin et al, 2019;Sharpe et al, 2017).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dopamine neuron ensembles signal the content of sensory prediction errors

Stalnaker

Howard²,

Takahashi

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Dopamine neurons respond to errors in predicting value-neutral sensory information.These data, combined with causal evidence that dopamine transients support sensory-based associative learning, suggest that the dopamine system signals a multidimensional prediction error. Yet such complexity is not evident in individual neuron or average neural activity. How then do downstream areas know what to learn in response to these signals? One possibility is that information about content is contained in the pattern of firing across many dopamine neurons. Consistent with this, here we show that the pattern of firing across a small group of dopamine neurons recorded in rats signals the identity of a mis-predicted sensory event.Further, this same information is reflected in the BOLD response elicited by sensory prediction errors in human midbrain. These data provide evidence that ensembles of dopamine neurons provide highly specific teaching signals, opening new possibilities for how this system might contribute to learning.

show abstract

Section: Discussionsupporting

confidence: 90%

Section: Introductionmentioning

confidence: 99%

Dopamine neuron ensembles signal the content of sensory prediction errors

Stalnaker

Howard²,

Takahashi

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recent work suggests that these representation-mediated phenomena are dopaminergically mediated. For example, behavioral over-expectations based on the rats' associatively learned model of reward contingencies are driven by dopamine signaling [94][95][96].…”

Section: Basic Preclinical Mechanismsmentioning

confidence: 99%

Hallucinations and Strong Priors

Corlett

Horga

Fletcher

et al. 2019

Trends in Cognitive Sciences

415

339

View full text Add to dashboard Cite

“…The striatum is hypothesized to receive reward prediction error (RPE) signals -- the difference between received and expected rewards -- from midbrain dopamine neurons (Barto, 1995; Montague et al, 1996; Schultz et al, 1997). The most common description of an RPE is as a “model-free” error, computed relative to the scalar value of a particular action, which itself reflects a common-currency based on a running average of previous rewards contingent on that action (Langdon et al, 2017). However, recent work suggests that RPE signals in the striatum can also reflect “model-based” information (Daw et al, 2011), where the prediction error is based on an internal simulation of future states.…”

Section: Introductionmentioning

confidence: 99%

Neural Signatures of Prediction Errors in a Decision-Making Task Are Modulated by Action Execution Failures

McDougle¹,

Butcher

Parvin³

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

20Decisions must be implemented through actions, and actions are prone to error. As such, when 21 an expected outcome is not obtained, an individual should not only be sensitive to whether the 22 choice itself was suboptimal, but also whether the action required to indicate that choice was 23 executed successfully. The intelligent assignment of credit to action execution versus action 24 selection has clear ecological utility for the learner. To explore this scenario, we used a modified 25 version of a classic reinforcement learning task in which feedback indicated if negative prediction 26 errors were, or were not, associated with execution errors. Using fMRI, we asked if prediction 27 error computations in the human striatum, a key substrate in reinforcement learning and decision 28 making, are modulated when a failure in action execution results in the negative outcome. 29Participants were more tolerant of non-rewarded outcomes when these resulted from execution 30 errors versus when execution was successful but the reward was withheld. Consistent with this 31 behavior, a model-driven analysis of neural activity revealed an attenuation of the signal 32 associated with negative reward prediction error in the striatum following execution failures. 33These results converge with other lines of evidence suggesting that prediction errors in the 34 mesostriatal dopamine system integrate high-level information during the evaluation of 35 instantaneous reward outcomes. 36 37 3 of 35 Introduction 38When a desired outcome is not obtained during instrumental learning, the agent should 39 be compelled to learn why. For instance, if an opposing player hits a home run, a baseball pitcher 40 needs to properly assign credit for the negative outcome: The error could have been in the decision 41 about the chosen action (e.g., throwing a curveball rather than a fastball) or the execution of that 42 decision (e.g., letting the curveball break over the plate rather than away from the hitter, as 43 planned). Here we ask if teaching signals in the striatum, a crucial region for reinforcement 44 learning, are sensitive to this dissociation. 45The striatum is hypothesized to receive reward prediction error (RPE) signals --the 46 difference between received and expected rewards --from midbrain dopamine neurons (Barto, 47 1995; Montague et al., 1996; Schultz et al., 1997). The most common description of an RPE is as a 48 "model-free" error, computed relative to the scalar value of a particular action, which itself reflects 49 a common-currency based on a running average of previous rewards contingent on that action 50 (Langdon et al., 2017). However, recent work suggests that RPE signals in the striatum can also 51 reflect "model-based" information (Daw et al., 2011), where the prediction error is based on an 52 internal simulation of future states. Moreover, human striatal RPEs have been shown to be 53 affected by a slew of cognitive factors, including attention (Leong et al., 2017), episodic memory 54 (Bornstein et al., 2017; Wimme...

show abstract

Model-based predictions for dopamine

Cited by 139 publications

References 69 publications

Dopamine neuron ensembles signal the content of sensory prediction errors

Dopamine neuron ensembles signal the content of sensory prediction errors

Hallucinations and Strong Priors

Neural Signatures of Prediction Errors in a Decision-Making Task Are Modulated by Action Execution Failures

Contact Info

Product

Resources

About