By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism

Frank, Michael J.; Seeberger, Lauren; O’Reilly, Randall C.

doi:10.1126/science.1102941

Cited by 1,790 publications

(2,357 citation statements)

References 34 publications

Supporting

176

Mentioning

2,142

Contrasting

Unclassified

Order By: Relevance

“…In particular, studies in patients with severe PD, accompanied by DA loss in the NAc, will reveal whether or not the L-DOPA-induced deficit in mild PD depends on the level of DA depletion in the NAc. Whereas other accounts of the medication-induced impairment do not require the NAc to be intact (Frank et al, 2004), we predict that the impairment is abolished with progression of the disease.…”

Section: Discussioncontrasting

confidence: 65%

“…In keeping with this prediction, the present observation confirms that L-DOPA interacts with the NAc, not the dorsal striatum during reversal learning. One mechanism by which L-DOPA may 'over-dose' reversal learning was proposed by Frank et al (2004). These authors suggested that L-DOPA-induced increases in tonic DA may 'fill in' phasic DA dips, thought to accompany omissions of reward (Hollerman and Schultz, 1998), thereby attenuating reward-prediction error signals.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

L-DOPA Disrupts Activity in the Nucleus Accumbens during Reversal Learning in Parkinson's Disease

Cools

Lewis

Clark

et al. 2006

Neuropsychopharmacol

268

210

View full text Add to dashboard Cite

Evidence indicates that dopaminergic medication in Parkinson's disease may impair certain aspects of cognitive function, such as reversal learning. We used functional magnetic resonance imaging in patients with mild Parkinson's disease to investigate the neural site at which L-DOPA acts during reversal learning. Patients were scanned both ON and OFF their normal dopamine-enhancing L-DOPA medication during the performance of a probabilistic reversal learning task. We demonstrate that L-DOPA modulated reversal-related activity in the nucleus accumbens, but not in the dorsal striatum or the prefrontal cortex. These data concur with evidence from studies with experimental animals and indicate an important role for the human nucleus accumbens in the dopaminergic modulation of reversal learning.

show abstract

Section: Discussioncontrasting

confidence: 65%

Section: Discussionmentioning

confidence: 99%

L-DOPA Disrupts Activity in the Nucleus Accumbens during Reversal Learning in Parkinson's Disease

Cools

Lewis

Clark

et al. 2006

Neuropsychopharmacol

268

210

View full text Add to dashboard Cite

show abstract

“…Here we report simulations with a simplified and generalized version of the probabilistic selection task (Frank, Moustafa, et al, 2007;Frank et al, 2004; see Figure 3), but the same results hold with the empirical version of the task.…”

Section: Probabilistic Selection Taskmentioning

confidence: 70%

“…Without the multiplicative update, G and N weights evolve symmetrically, with no preferential differentiation among positive or negative values, and as such are linear combinations of true expected value, leading to equal Choose-A and Avoid-B performance (see Appendix, supplemental simulations in Figure A2). Dopaminergic manipulations in the probabilistic selection task have been shown repeatedly (Frank, Moustafa, et al, 2007;Frank & O'Reilly, 2006;Frank et al, 2004;Jocham et al, 2011;Shiner et al, 2012;Smittenaar et al, 2012) to induce changes in Choose-A versus Avoid-B bias, although it has not been clearly disentangled whether this was due to learning effects or performance effects. As noted earlier, some studies show effects of dopamine medication in which Choose-A performance is improved even when the design was such that medications could have only affected test performance rather than learning (Shiner et al, 2012;Smittenaar et al, 2012).…”

mentioning

confidence: 99%

Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.

Collins¹,

Frank²

2014

Psychological Review

Self Cite

391

455

View full text Add to dashboard Cite

The striatal dopaminergic system has been implicated in reinforcement learning (RL), motor performance, and incentive motivation. Various computational models have been proposed to account for each of these effects individually, but a formal analysis of their interactions is lacking. Here we present a novel algorithmic model expanding the classical actor-critic architecture to include fundamental interactive properties of neural circuit models, incorporating both incentive and learning effects into a single theoretical framework. The standard actor is replaced by a dual opponent actor system representing distinct striatal populations, which come to differentially specialize in discriminating positive and negative action values. Dopamine modulates the degree to which each actor component contributes to both learning and choice discriminations. In contrast to standard frameworks, this model simultaneously captures documented effects of dopamine on both learning and choice incentive-and their interactions-across a variety of studies, including probabilistic RL, effort-based choice, and motor skill learning.Keywords: dopamine, striatum, reinforcement learning, choice incentive, computational model Dopamine plays a crucial role in human and animal cognition, substantially influencing a diversity of processes including reinforcement learning, motivation, incentive, working memory, and effort. Dopaminergic neurons in the substantial nigra and ventral tegmental area project to a very wide set of subcortical and cortical areas, with strongest innervation in the basal ganglia (BG), specifically in the ventral and dorsal striatum. Dysregulation of dopamine is present in a wide array of mental illnesses such as Parkinson's disease, attention-deficit/hyperactivity disorder (ADHD), schizophrenia, and Tourette's syndrome and is a central pharmaceutical target used to treat symptoms across these and numerous other pathologies.Although considerable progress has been made in our understanding of its various distinct roles, there remain fundamental debates concerning its precise mechanisms and functions, especially regarding their integration and interactions. In reward-based decision making in particular, two largely separate traditions have studied the reinforcement learning and the incentive theories of dopamine (Berridge, 2007). Despite solid evidence for both theories, theoretical and empirical studies tend to favor and focus on one or the other interpretation, with little attempt to unify them or to study their interaction. Here we develop an explicit computational analysis of the dual role of striatal dopamine in modulation of incentive motivation (affecting choice), reinforcement learning, and how these processes interact. This endeavor allows us not only to account for both types of findings alone but also those that could not be explained by either theory in isolation. RL Theory of DopamineOne widely accepted theory of dopamine function relates to its role in model-free reinforcement learning (RL). Specifically, phasic f...

show abstract

“…It is not clear whether the reduction seen in addiction is mainly pre-or postsynaptic, but both could potentially promote drug-taking. Postsynaptically, they have been shown to mediate the effect of losses on "go/no-go" learning (Frank, 2005;Frank et al, 2004;Dreyer et al, 2010;Kravitz et al, 2012), and could thereby contribute to the insensitivity towards adverse consequences in addiction (Vanderschuren and Everitt, 2004;Deroche-Gamonet et al, 2004;Maia and Frank, 2011;Kravitz et al, 2012). Presynaptically, they are involved in an autoinhibitory negative feedback loop which could particularly affect go-learning as it could reduce the positive phasic transients (Bello et al, 2011) and thereby lead to the sort of increased prediction error increase mentioned above (Bello et al 2011; see also Sulzer 2011).…”

Section: Phasic Dopaminergic Signals In Addictionmentioning

confidence: 99%

The role of learning-related dopamine signals in addiction vulnerability

Huys

Tobler

Hasler

et al. 2014

Progress in Brain Research

View full text Add to dashboard Cite

Psychostimulants such as methylphenidate (MPH) and antidepressants such as fluoxetine (FLX) are widely used in the treatment of various mental disorders or as cognitive enhancers. These medications are often combined, for example, to treat comorbid disorders. There is a considerable body of evidence from animal models indicating that individually these psychotropic medications can have detrimental effects on the brain and behavior, especially when given during sensitive periods of brain development. However, almost no studies investigate possible interactions between these drugs. This is surprising given that their combined neurochemical effects (enhanced dopamine and serotonin neurotransmission) mimic some effects of illicit drugs such as cocaine and amphetamine. Here, we summarize recent studies in juvenile rats on the molecular effects in the mid-and forebrain and associated behavioral changes, after such combination treatments. Our findings indicate that these combined MPH + FLX treatments can produce similar molecular changes as seen after cocaine exposure while inducing behavioral changes indicative of dysregulated mood and motivation, effects that often endure into adulthood. Tables: 2 References: 254 ABSTRACT Dopaminergic signals play a mathematically precise role in reward-related learning, and variations in dopaminergic signalling have been implicated in vulnerability to addiction. Here, we provide a detailed overview of the relationship between theoretical, mathematical and experimental accounts of phasic dopamine signalling, with implications for the role of learning-related dopamine signalling in addiction and related disorders. We describe the theoretical and behavioural characteristics of model-free learning based on errors in the prediction of reward, including step-by-step explanations of the underlying equations. We then use recent insights from an animal model that highlights individual variation in learning during a Pavlovian conditioning paradigm to describe overlapping aspects of incentive salience attribution and model-free learning. We argue that this provides a computationally coherent account of some features of addiction. CONTENTS

show abstract

By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism

Cited by 1,790 publications

References 34 publications

L-DOPA Disrupts Activity in the Nucleus Accumbens during Reversal Learning in Parkinson's Disease

L-DOPA Disrupts Activity in the Nucleus Accumbens during Reversal Learning in Parkinson's Disease

Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.

The role of learning-related dopamine signals in addiction vulnerability

Contact Info

Product

Resources

About