Dopamine transients delivered in learning contexts do not act as model-free prediction errors

Abstract

Dopamine neurons fire transiently in response to unexpected rewards. These neural correlates are proposed to signal the reward prediction error described in model-free reinforcement learning algorithms. This error term represents the unpredicted or 'excess' value of the rewarding event. In model-free reinforcement learning, this value is then stored as part of the learned value of any antecedent cues, contexts or events, making them intrinsically valuable, independent of the specific rewarding event that caused the prediction error. In support of equivalence between dopamine transients and this model-free error term, proponents cite causal optogenetic studies showing that artificially induced dopamine transients cause lasting changes in behavior. Yet none of these studies directly demonstrate the presence of cached value under conditions appropriate for associative learning. To address this gap in our knowledge, we conducted three studies where we optogenetically activated dopamine neurons while rats were learning associative relationships, both with and without reward. In each experiment, the antecedent cues failed to acquired value and instead entered into value-independent associative relationships with the other cues or rewards. These results show that dopamine transients, constrained within appropriate learning situations, support valueless associative learning.

Substantial decline of phasic dopamine signaling in senescent male rats does not impact dopamine-dependent Pavlovian conditioning

“…In the model-free system, cache values are assigned to appetitive cues, values that in turn are updated through experience via mechanisms thought to follow temporal difference learning algorithms (Sutton and Barto, 1987;Schultz et al, 1997) based upon formal learning theories (Rescorla Wagner, 1972). Previous work has identified dopamine signaling as a potential neural substrate for the model-free system and, in particular, phasic dopamine signaling (Flagel, Clark et al, 2011), has been implicated in aspects of associative learning and decision making (Gan et al, 2010;Saunders et al, 2018;Sharpe et al, 2020). Moreover, other studies have shown biochemical (Morgan et al, 1987;Moretti et al, 1987;Aria and Kinemuchi, 1988;Gozlan et al, 1990;Suzuki et al, 2001), neurochemical (Yurek et al, 1998;Friedemann and Gerhardt, 1992) evidence indicating a decline in dopamine signaling associated with senescence.…”

Disruptions in effort-based decision-making following acute optogenetic stimulation of ventral tegmental area dopamine cells

“…Accordingly, in the optogenetic stimulation sessions decision-making performance might be influenced by disruptions in associative mechanisms controlled by reward prediction error signals. Furthermore, more recent studies suggest dopamine transients can be uncoupled from model-free reinforcement value signals to encode associative information that is computationally detailed in nature (e.g., its sensory features; Gardner et al 2018 ; Sharpe et al 2019 ). With this in mind, the current findings also suggest that dopamine functions beyond a pure value-based signal, to include recall of past, and/or anticipation of detailed upcoming contingencies, which may bias performance toward actions that overall have the highest immediacy of reinforcer delivery.…”

Disruptions in effort-based decision-making following acute optogenetic stimulation of ventral tegmental area dopamine cells

“…Furthermore, more recent studies suggest dopamine transients can be uncoupled from model-free reinforcement value signals to encode associative information that is computationally detailed in nature (e.g. its sensory features; Gardner et al, 2018;Sharpe et al, 2019). This in mind, the current findings also suggest that dopamine functions beyond a pure value-based signal, to include recall of past, and/or anticipation of detailed upcoming contingencies, which may bias performance towards actions that overall have the highest immediacy of reinforcer delivery.…”