Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning

O’Doherty, John P.; Dayan, Peter; Schultz, Johannes; Deichmann, Ralf; Friston, Karl J.; Dolan, Raymond J.

doi:10.1126/science.1094285

Cited by 1,884 publications

(1,630 citation statements)

References 26 publications

Supporting

162

Mentioning

1,458

Contrasting

Unclassified

Order By: Relevance

“…More specifically, changes in state values V MF (s) imply changes in future reward, and so a change in value induced by an action is a metric that can be used to reinforce behaviours. This forms the core of the actor-critic model (Barto et al, 1983;O'Doherty et al, 2004). Experimentally, it is perhaps most directly demonstrated by conditioned reinforcement experiments (Everitt and Robbins, 2005;Meyer et al, 2012), where instrumental behaviours can be reinforced by Pavlovian CSs.…”

Section: Instrumental Behaviourmentioning

confidence: 98%

The role of learning-related dopamine signals in addiction vulnerability

Huys

Tobler

Hasler

et al. 2014

Progress in Brain Research

View full text Add to dashboard Cite

Psychostimulants such as methylphenidate (MPH) and antidepressants such as fluoxetine (FLX) are widely used in the treatment of various mental disorders or as cognitive enhancers. These medications are often combined, for example, to treat comorbid disorders. There is a considerable body of evidence from animal models indicating that individually these psychotropic medications can have detrimental effects on the brain and behavior, especially when given during sensitive periods of brain development. However, almost no studies investigate possible interactions between these drugs. This is surprising given that their combined neurochemical effects (enhanced dopamine and serotonin neurotransmission) mimic some effects of illicit drugs such as cocaine and amphetamine. Here, we summarize recent studies in juvenile rats on the molecular effects in the mid-and forebrain and associated behavioral changes, after such combination treatments. Our findings indicate that these combined MPH + FLX treatments can produce similar molecular changes as seen after cocaine exposure while inducing behavioral changes indicative of dysregulated mood and motivation, effects that often endure into adulthood. Tables: 2 References: 254 ABSTRACT Dopaminergic signals play a mathematically precise role in reward-related learning, and variations in dopaminergic signalling have been implicated in vulnerability to addiction. Here, we provide a detailed overview of the relationship between theoretical, mathematical and experimental accounts of phasic dopamine signalling, with implications for the role of learning-related dopamine signalling in addiction and related disorders. We describe the theoretical and behavioural characteristics of model-free learning based on errors in the prediction of reward, including step-by-step explanations of the underlying equations. We then use recent insights from an animal model that highlights individual variation in learning during a Pavlovian conditioning paradigm to describe overlapping aspects of incentive salience attribution and model-free learning. We argue that this provides a computationally coherent account of some features of addiction. CONTENTS

show abstract

Section: Instrumental Behaviourmentioning

confidence: 98%

The role of learning-related dopamine signals in addiction vulnerability

Huys

Tobler

Hasler

et al. 2014

Progress in Brain Research

View full text Add to dashboard Cite

show abstract

“…Actions that produce positive prediction errors are reinforced, whereas those that produce negative prediction errors are punished. Commonly, the critic is assigned to ventral striatum and the amygdala (Hazy, Frank, & O'Reilly, 2010;O'Doherty et al, 2004), whereas the actor is considered to be instantiated by dorsal striatal interactions with pre/motor cortex.Critic learning. The critic in our model is similar to that in classical formulations (but see Discussion).…”

mentioning

confidence: 99%

Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.

Collins¹,

Frank²

2014

Psychological Review

390

455

View full text Add to dashboard Cite

The striatal dopaminergic system has been implicated in reinforcement learning (RL), motor performance, and incentive motivation. Various computational models have been proposed to account for each of these effects individually, but a formal analysis of their interactions is lacking. Here we present a novel algorithmic model expanding the classical actor-critic architecture to include fundamental interactive properties of neural circuit models, incorporating both incentive and learning effects into a single theoretical framework. The standard actor is replaced by a dual opponent actor system representing distinct striatal populations, which come to differentially specialize in discriminating positive and negative action values. Dopamine modulates the degree to which each actor component contributes to both learning and choice discriminations. In contrast to standard frameworks, this model simultaneously captures documented effects of dopamine on both learning and choice incentive-and their interactions-across a variety of studies, including probabilistic RL, effort-based choice, and motor skill learning.Keywords: dopamine, striatum, reinforcement learning, choice incentive, computational model Dopamine plays a crucial role in human and animal cognition, substantially influencing a diversity of processes including reinforcement learning, motivation, incentive, working memory, and effort. Dopaminergic neurons in the substantial nigra and ventral tegmental area project to a very wide set of subcortical and cortical areas, with strongest innervation in the basal ganglia (BG), specifically in the ventral and dorsal striatum. Dysregulation of dopamine is present in a wide array of mental illnesses such as Parkinson's disease, attention-deficit/hyperactivity disorder (ADHD), schizophrenia, and Tourette's syndrome and is a central pharmaceutical target used to treat symptoms across these and numerous other pathologies.Although considerable progress has been made in our understanding of its various distinct roles, there remain fundamental debates concerning its precise mechanisms and functions, especially regarding their integration and interactions. In reward-based decision making in particular, two largely separate traditions have studied the reinforcement learning and the incentive theories of dopamine (Berridge, 2007). Despite solid evidence for both theories, theoretical and empirical studies tend to favor and focus on one or the other interpretation, with little attempt to unify them or to study their interaction. Here we develop an explicit computational analysis of the dual role of striatal dopamine in modulation of incentive motivation (affecting choice), reinforcement learning, and how these processes interact. This endeavor allows us not only to account for both types of findings alone but also those that could not be explained by either theory in isolation. RL Theory of DopamineOne widely accepted theory of dopamine function relates to its role in model-free reinforcement learning (RL). Specifically, phasic f...

show abstract

“…Neuroimaging studies of reward processing have identified a number of brain areas that are activated by the delivery of primary reinforcers such as appetitive stimuli (Berns et al, 2001;McClure et al, 2003;O'Doherty et al, 2004), and secondary reinforcement such as monetary gains and losses (Breiter et al, 2001;Delgado et al, 2004;Elliott et al, 2000;Holroyd et al, 2004b;Thut et al, 1997). However, it remains to be determined precisely how information about reward and punishment is encoded in these reward-sensitive neural circuits.…”

mentioning

confidence: 99%

Activity in human reward-sensitive brain areas is strongly context dependent

et al. 2005

View full text Add to dashboard Cite

Functional neuroimaging research in humans has identified a number of brain areas that are activated by the delivery of primary and secondary reinforcers. The present study investigated how activity in these reward-sensitive regions is modulated by the context in which rewards and punishments are experienced. Fourteen healthy volunteers were scanned during the performance of a simple monetary gambling task that involved a bwinQ condition (in which the possible outcomes were a large monetary gain, a small gain, or no gain of money) and a bloseQ condition (in which the possible outcomes were a large monetary loss, a small loss, or no loss of money). We observed reward-sensitive activity in a number of brain areas previously implicated in reward processing, including the striatum, prefrontal cortex, posterior cingulate, and inferior parietal lobule. Critically, activity in these reward-sensitive areas was highly sensitive to the range of possible outcomes from which an outcome was selected. In particular, these regions were activated to a comparable degree by the best outcomes in each condition-a large gain in the win condition and no loss of money in the lose condition-despite the large difference in the objective value of these outcomes. In addition, some rewardsensitive brain areas showed a binary instead of graded sensitivity to the magnitude of the outcomes from each distribution. These results provide important evidence regarding the way in which the brain scales the motivational value of events by the context in which these events occur. D

show abstract

Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning

Cited by 1,884 publications

References 26 publications

The role of learning-related dopamine signals in addiction vulnerability

The role of learning-related dopamine signals in addiction vulnerability

Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.

Activity in human reward-sensitive brain areas is strongly context dependent

Contact Info

Product

Resources

About