Reward prediction-errors weighted by cue salience produces addictive behaviors in simulations, with asymmetrical learning and steeper delay discounting

Abstract

Dysfunction in learning and motivational systems are thought to contribute to addictive behaviors. Previous models have suggested that dopaminergic roles in learning and motivation could produce addictive behaviors through pharmacological manipulations that provide excess dopaminergic signaling towards these learning and motivational systems. Redish 2004 suggested a role based on dopaminergic signals of value prediction error, while Zhang et al. 2009 suggested a role based on dopaminergic signals of motivation. Both these models present significant limitations. They do not explain the reduced sensitivity to drug-related costs/negative consequences, the increased impulsivity generally found in people with a substance use disorder, craving behaviors, and non-pharmacological dependence, all of which are key hallmarks of addictive behaviors. Here, we propose a novel mathematical definition of salience, that combines aspects of dopaminergic function in both, learning and motivation, within the reinforcement learning framework. Using a single parameter regime, we simulated addictive behaviors that the Zhang et al. 2009 and Redish 2004 models also produce but we went further in simulating the downweighting of drug-related negative prediction-errors, steeper delay discounting of drug rewards, craving behaviors and aspects of behavioral/non-pharmacological addictions. The current salience model builds on our recently proposed conceptual theory that salience modulates internal representation updating and may contribute to addictive behaviors by producing misaligned internal representations (Kalhan et al., 2021). Critically, our current mathematical model of salience argues that the seemingly disparate learning and motivational aspects of dopaminergic functioning may interact through a salience mechanism that modulates internal representation updating.

Electronic gambling machines and dopamine.

“…Saliency signals in the paraventricular thalamus affect associative learning [96], possibly via a modulation prediction error signaling in the VTA [96,97]. This raises the possibility of interactions between prediction error and saliency signaling at the level of midbrain dopaminergic circuits, in line with recent theoretical models [98]. Saliency might also bias the encoding of specific events via interactions of the DA system with the hippocampus [99][100][101].…”

“…For example, as noted above, both prediction error and uncertainty effects in EGM gambling co-occur with perceptual saliency effects, raising the potential for interactive effects [98]. In line with this idea, inherent cue characteristics can modulate human reinforcement learning [103], resonating with the possibility that saliency potentiates prediction error effects [96,98]. A unique example demonstrating such interactive effects in humans comes from a study by Cherkasova and colleagues [93], showing that sensory cues presented along with winning outcomes in a risk-taking task increase risk-taking behavior.…”

“…One of several central open questions (see Outstanding Questions) concerns potential interactions amongst these factors. For example, as noted above, both prediction error and uncertainty effects in EGM gambling co-occur with perceptual saliency effects, raising the potential for interactive effects [98]. In line with this idea, inherent cue characteristics can modulate human reinforcement learning [103], resonating with the possibility that saliency potentiates prediction error effects [96,98].…”