Dopamine Neurons Respond to Errors in the Prediction of Sensory Features of Expected Rewards

Abstract

Summary Midbrain dopamine neurons have been proposed to signal prediction errors as defined in model-free reinforcement learning algorithms. While these algorithms have been extremely powerful in interpreting dopamine activity, these models do not register any error unless there is a difference between the value of what is predicted and what is received. Yet learning often occurs in response to changes in the unique features that characterize what is received, sometimes with no change in its value at all. Here, we show that classic error-signaling dopamine neurons also respond to changes value-neutral sensory features of an expected reward. This suggests that dopamine neurons have access to a wider variety of information than contemplated by the models currently used to interpret their activity and that, while their firing may conform to predictions of these models in some cases, they are not restricted to signaling errors in the prediction of value.

Orbitofrontal neurons signal reward predictions, not reward prediction errors

“…2A and B) was almost the inverse of that of DA neurons recorded in the same task (Takahashi et al, 2017). In our hands in this task, DA neurons begin to respond about 100 ms after reward delivery and peak 200–300 ms after each reward drop, whereas the OFC neurons began to respond before each drop of reward, peaked about 100 ms after delivery, and reached a local minimum 250–300 ms after reward delivery.…”

“…Previously we have described how neurons in this dataset fired to the predictive cues (Stalnaker et al, 2014); here we focused on the firing in anticipation of and just after delivery of each drop of reward, looking specifically for differences in firing based on whether the reward number or flavor was unexpected or expected. Unlike dopamine neurons recorded in this setting, which exhibited phasic error-like responses after surprising changes in either reward number or reward flavor (Takahashi et al, 2017), OFC neurons showed no such error correlates and instead fired in a way that reflected reward predictions. …”

“…We compared these data with recordings from dopamine neurons recorded in rats in the same task, as reported earlier (Takahashi et al, 2017). Figures from that dataset (insets in Figs.…”

Evaluation of the hypothesis that phasic dopamine constitutes a cached-value signal

“…That is, if a cue previously paired with a particular reward was unexpectedly presented with a different reward that was equally valuable, we would expect to see a prediction error in dopaminergic neurons. And in fact recent evidence has emerged to suggest dopamine does in fact encode such information (Takahashi et al, 2017). Specifically, Takahashi et al (2017) have shown that dopamine neurons exhibit their classic prediction-error signal to changes in the sensory properties of rewards that are equally prefered.…”

“…And in fact recent evidence has emerged to suggest dopamine does in fact encode such information (Takahashi et al, 2017). Specifically, Takahashi et al (2017) have shown that dopamine neurons exhibit their classic prediction-error signal to changes in the sensory properties of rewards that are equally prefered. That is, dopamine neurons show errors to the change in reward identity without a change in reward value.…”