A Behavioral Association Between Prediction Errors and Risk-Seeking: Theory and Evidence

Abstract

9Reinforcement learning theories propose that humans choose based on the estimated values of 10 available options, and that they learn from rewards by reducing the difference between the experienced 11 and expected value. In the brain, such prediction errors are broadcasted by dopamine. However, choices 12 are not only influenced by expected value, but also by risk. Like reinforcement learning, risk preferences 13 are modulated by dopamine: enhanced dopamine levels induce risk-seeking. Learning and risk 14 preferences have so far been studied independently, even though it is commonly assumed that they are 15 (partly) regulated by the same neurotransmitter. Here, we use a novel learning task to look for 16 prediction-error induced risk-seeking in human behavior and pupil responses. We find that prediction 17 errors are positively correlated with risk-preferences in imminent choices. Physiologically, this effect is 18 indexed by pupil dilation: only participants whose pupil response indicates that they experienced the 19 prediction error also show the behavioral effect. 20Reward-guided learning in humans and animals can often be modelled simply as reducing the difference 22 between the obtained and expected reward-a reward prediction error. This well-established 23 behavioral phenomenon [Rescorla, 1972] has been linked to the neurotransmitter dopamine [Schultz, 24 1997]. It has been shown that bursts of dopaminergic activity broadcast prediction errors to brain areas 25 that are relevant for reward learning, such as the striatum, the amygdala, and the prefrontal cortex. 26Another behavioral phenomenon that has been well studied is the effect of uncertainty and risk on 27 decision making [Kahneman, 2013]. Here again, a different line of research has established an 28 association between dopamine and risk-taking: dopamine-enhancing medication has been shown to 29 increase risk-seeking in rats [St Onge, 2009], and drive excessive gambling when used to treat 30Parkinson's disease [Voon, 2006] [Gallagher, 2007] [Weintraub, 2010]. More recently, it has been 31 demonstrated that phasic responses in dopaminergic brain areas modulate moment-by-moment risk-32 preference in humans: the tendency to take risks correlated positively with the magnitude of task-33 related dopamine responses [Chew, 2019]. A family of mechanistic theories of the basal ganglia network 34 provides an explanation for these risk effects [Mikhael, 2016] [Moeller, 2019]. According to these 35 models, positive and negative outcomes of actions are encoded separately in direct and indirect 36 pathways of the basal ganglia. The balance between those pathways is controlled by the dopamine 37level. An increased dopamine level promotes the direct pathway and hence puts emphasis on potential 38 gains, thus rendering risky options more attractive. 39In summary, dopamine bursts are related to distinct behavioral phenomena-learning and risk-taking-40 by way of 1) acting as reward prediction errors, affecting synaptic weights during reinforcement 41 learnin...

Uncertainty-guided learning with scaled prediction errors in the basal ganglia

“…However, it has been recently demonstrated that even a brief, burst-like activation of dopaminergic neurons changes the activity levels of striatal neurons (31). Additionally, it has been shown that reward prediction errors modulate the tendency to make risky choices (22), and risk attitudes are known to depend on the balance between the direct and indirect pathways (32, 33). In this paper, we demonstrated that a more realistic assumption, that the dopamine signal encoding prediction error also changes the activity levels in striatum, enables scaling of prediction errors by uncertainty.…”

“…In Eq. 7 and 8, 𝐺 and 𝑁 denote the synaptic inputs in the direct and indirect pathway respectively (22), and 𝜆 is a coefficient determining the accuracy with which the standard deviation can be encoded (as explained below). These assumptions can be used to rewrite the learning rules given in Eq.…”