Evaluation of the hypothesis that phasic dopamine constitutes a cached-value signal

Abstract

The phasic dopamine error signal is currently argued to be synonymous with the prediction error in Sutton and Barto (1987, 1998) model-free reinforcement learning algorithm (Schultz et al., 1997). This theory argues that phasic dopamine reflects a cached-value signal that endows reward-predictive cues with the scalar value inherent in reward. Such an interpretation does not envision a role for dopamine in more complex cognitive representations between events which underlie many forms of associative learning, restricting the role dopamine can play in learning. The cached-value hypothesis of dopamine makes three concrete predictions about when a phasic dopamine response should be seen and what types of learning this signal should be able to promote. We discuss these predictions in light of recent evidence which we believe provide particularly strong tests of their validity. In doing so, we find that while the phasic dopamine signal conforms to a cached-value account in some circumstances, other evidence demonstrate that this signal is not restricted to a model-free cached-value reinforcement learning signal. In light of this evidence, we argue that the phasic dopamine signal functions more generally to signal violations of expectancies to drive real-world associations between events.

Sensory prediction errors in the human midbrain signal identity violations independent of perceptual distance

“…In addition, identity PEs in the midbrain correlate with updates of outcome identity expectations in the orbitofrontal cortex (OFC) (Howard and Kahnt, 2018). This suggests a broader role for dopamine in associative learning, involving features beyond value (Gardner et al, 2018; Sharpe and Schoenbaum, 2018).…”

“…Seminal studies have shown that dopaminergic neurons in the midbrain display firing patterns consistent with these errors, which have been taken as a neural substrate of reward learning (Schultz et al, 1997; Schultz et al, 2017). Interestingly, unlike the original formulation of PEs as a teaching signal that is agnostic to the type of information that is encoded, dopaminergic PE responses have been conceptualized to exclusively respond to the difference between the value of expected and received outcomes (Glimcher, 2011; Sharpe and Schoenbaum, 2018).…”

Dynamic changes in Anterior Cingulate Cortex ensembles mark the transition from exploration to exploitation

“…ACC neurons and ensembles also encode a regency-weighted history of past outcomes (Seo & Lee 2007;Bernacchia et al 2011;Hyman et al 2017) and it has been argued that these signals might produce the value representations that ultimately guide foraging decisions Wittmann et al 2016). The idea that action representations accrue value through an experience-dependent process is consistent with theories of reinforcement learning (RL) in mesolimbic pathways where value representations are thought to be continuously updated through a dopamine-dependent process (Stauffer et al 2014;Schultz 2015;Kolling & Akam 2017;Sharpe & Schoenbaum 2018).…”

Ramping and phasic dopamine activity accounts for efficient cognitive resource allocation during reinforcement learning

“…Overall, our results show that transitions from phasic to ramping activity reflect widespread resource distribution, which allow rapid adaptation to potential changes in the stimulus value. Recent studies have found that the dopamine neurons are responsive to novel stimuli [60,61], changes to reward features [62,63], and stimuli that are potentially relevant to a given task [2][3][4][64][65][66].…”