A Role for Dopamine in Temporal Decision Making and Reward Maximization in Parkinsonism

Abstract

Converging evidence implicates striatal dopamine (DA) in reinforcement learning, such that DA increases enhance "Go learning" to pursue actions with rewarding outcomes, whereas DA decreases enhance "NoGo learning" to avoid non-rewarding actions. Here we test whether these effects apply to the response time domain. We employ a novel paradigm which requires the adjustment of response times to a single response. Reward probability varies as a function of response time, whereas reward magnitude changes in the opposite direction. In the control condition, these factors exactly cancel, such that the expected value across time is constant (CEV). In two other conditions, expected value increases (IEV) or decreases (DEV), such that reward maximization requires either speeding up (Go learning) or slowing down (NoGo learning) relative to the CEV condition. We tested patients with Parkinson's disease (depleted striatal DA levels) on and off dopaminergic medication, compared with age-matched controls. While medicated, patients were better at speeding up in the DEV relative to CEV conditions. Conversely, nonmedicated patients were better at slowing down to maximize reward in the IEV condition. These effects of DA manipulation on cumulative Go/NoGo response time adaptation were captured with our a priori computational model of the basal ganglia, previously applied only to forced-choice tasks. There were also robust trial-to-trial changes in response time, but these single trial adaptations were not affected by disease or medication and are posited to rely on extrastriatal, possibly prefrontal, structures.

Computational models of reinforcement learning: the role of dopamine as a reward signal

“…Others have used a similar approach to show that these learning rate parameters are differentially modulated by dopaminergic medications administered to patients with Parkinson's (Voon et al 2010). Finally, another recent study replicated the striatal D1/D2 genetic variation effects on positive and negative learning rates in the context of a task that required participants to adjust response times to maximize rewards, which had also been shown to be sensitive to dopaminergic manipulation (Moustafa et al 2008;Frank et al 2009). Moreover, in addition to learning, this model also included a term to account for exploration, which was predicted to occur when participants were particularly uncertain about the reward statistics due to insufficient sampling.…”

“…Dopamine has been shown to play a role in motivation (Fibiger and Phillips 1986;Robbins and Everitt 1996;Wise and Hoffman 1992;Wise 2004Wise , 2005, in the acquisition of appetitive conditioning tasks (Berridge and Robinson 1998;Everitt et al 1999;Ikemoto and Panksepp 1999), in many aspects of drug addiction (Berke and Hyman 2000;Di Chiara 2002;Everitt and Robbins 2005;Kelley and Berridge 2002;Koob 1992;Robinson and Berridge 2008;Wise 1996a, b) and is involved in disorders such as Parkinson disease (Canavan et al 1989;Voon et al 2010;Frank 2005;Knowlton et al 1996;Moustafa et al 2008).…”

Neurogenetics and Pharmacology of Learning, Motivation, and Cognition

“…Recent feats in genetic engineering revealed striking support for these mechanisms, whereby reward and aversive/avoidance learning was impaired in animals with selected targeted disruption of striatonigral and striatopallidal cells, respectively (Hikida et al, 2010). Moreover, the model correctly predicted that human Parkinson's patients would show impairments in learning to make responses associated with positive outcomes but relative enhancements in NoGo learning from negative prediction errorsFand that both these effects would be reversed when patients were taking dopamine medication (Frank et al, 2004(Frank et al, , 2007bMoustafa et al, 2008a;Cools et al, 2006;Bodi et al, 2009;Palminteri et al, 2009;Voon et al, 2010).…”

“…First, faster responses are made as a function of the relative difference in activity levels between striatonigral/Go and striatopallidal/NoGo cells coding for the executed action. These relative activation differences can arise either because of previous learning (such that responses with higher reinforcement probabilities are more swiftly executed), or because of greater induced DA release (such that DA bursts as a function of novelty or reward predicting variables may directly influence Go vs NoGo activity, and hence response speed), or both (Moustafa et al, 2008a;Wiecki et al, 2009). It is to be noted that the same Go-NoGo mechanisms, operating in the ventral striatum as opposed to dorsal striatum coding of specific instrumental actions, can support the selection of general Pavlovian approach 'actions' in pursuit of rewards.…”

“…Conversely, enhanced striatal D2 receptor affinity in DRD2 C957T T/T carriers was associated with relatively greater learning from negative outcomes . Dopaminergic medication increased relative learning from positive outcomes in Parkinson's patients performing the same task (Moustafa et al, 2008a, where here we fit the mathematical model from to choices in the 2008 study). (b) Model fits enable inference regarding the learned 'Q' values of stimulus-action combinations as a function of their true probability of reinforcement.…”