The Dopamine Prediction Error: Contributions to Associative Models of Reward Learning

Abstract

Phasic activity of midbrain dopamine neurons is currently thought to encapsulate the prediction-error signal described in Sutton and Barto’s (1981) model-free reinforcement learning algorithm. This phasic signal is thought to contain information about the quantitative value of reward, which transfers to the reward-predictive cue after learning. This is argued to endow the reward-predictive cue with the value inherent in the reward, motivating behavior toward cues signaling the presence of reward. Yet theoretical and empirical research has implicated prediction-error signaling in learning that extends far beyond a transfer of quantitative value to a reward-predictive cue. Here, we review the research which demonstrates the complexity of how dopaminergic prediction errors facilitate learning. After briefly discussing the literature demonstrating that phasic dopaminergic signals can act in the manner described by Sutton and Barto (1981), we consider how these signals may also influence attentional processing across multiple attentional systems in distinct brain circuits. Then, we discuss how prediction errors encode and promote the development of context-specific associations between cues and rewards. Finally, we consider recent evidence that shows dopaminergic activity contains information about causal relationships between cues and rewards that reflect information garnered from rich associative models of the world that can be adapted in the absence of direct experience. In discussing this research we hope to support the expansion of how dopaminergic prediction errors are thought to contribute to the learning process beyond the traditional concept of transferring quantitative value.

Model-based fMRI Reveals Dissimilarity Processes Underlying Base Rate Neglect

“…This novelty-sensitive network included clusters in dorsomedial PFC, dorsolateral PFC, and superior parietal lobule. Alternatively, exposure to previously learned pairs at test was accompanied by activation of vmPFC, which has been associated with attention to strong predictors (Sharpe and Killcross, 2015;Nasser et al, 2017), high confidence / high relative decision evidence (Schnyer, Nicholls, and Verfaille, 2005;Lebreton et al, 2015;Davis et al, 2017) and the application of familiar rules (Boettiger and D'Esposito, 2005). Thus the current mean BOLD activation results are in accordance with a number of recent studies examining decision-related processes in categorization.…”

“…It is therefore possible that a rapid form of attention focusing on predictive utility was present during our experiment, with this effect being washed out by the subsequent effects of controlled attention on the ambiguous trials. Consistent with this idea, we found medial prefrontal regions associated with attention to strong predictors (Sharpe and Killcross, 2015;Nasser et al, 2017) to be engaged when participants encountered previously-. CC-BY-NC-ND 4.0 International license It is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.…”

Ventral Tegmental Dopamine Neurons Participate in Reward Identity Predictions

“…The role of DA teaching signals in value and identity learning remains unclear [16, 17]. Since the original discovery that they track changes in expected value, phasic dopamine signals have predominantly been interpreted as model-free RPEs, promoting pure value assignment.…”

A roadmap to integrate astrocytes into Systems Neuroscience

“…In predictive coding, neuromodulation is proposed as computing part of the statistics of errors made by predictions (Lau, Monteiro, & Paton, 2017;Stephan, Iglesias, Heinzle, & Diaconescu, 2015). The bulk of empirical support for predictive coding lies in the domains of perception, reward learning, and decision making, as documented in humans, monkeys, and rodents (Diederen et al, 2017;Kok & de Lange, 2014;Leinweber, Ward, Sobczak, Attinger, & Keller, 2017;Markov et al, 2014;Nasser, Calu, Schoenbaum, & Sharpe, 2017;Summerfield, Trittschuh, Monti, Mesulam, & Egner, 2008;Wacongne et al, 2011), whereas the framework appears to be under exploration in memory consolidation (Cross, Kohler, Schlesewsky, Gaskell, & Bornkessel-Schlesewsky, 2018) and emotion (Barrett, 2017). Other general CNS frameworks worth mentioning are global workspace theory, which describes the basic circuit from which consciousness emerges (Baars, 2005), and liquid computing, which states that neural circuits have the capacity to store information of previous perturbation(s), analogous to the ripples generated on the surface of a pond when stones are thrown into it (Maass, Natschlager, & Markram, 2002).…”