A Local Circuit Model of Learned Striatal and Dopamine Cell Responses under Probabilistic Schedules of Reward

Tan, Can Ozan; Bullock, Daniel

doi:10.1523/jneurosci.0259-08.2008

Cited by 39 publications

(61 citation statements)

References 81 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Biologically more plausible TD implementations and spectral timing models use stimulus eligibility traces, as originally suggested (FIGURE 14B) (574), and model well the single step backward transition of dopamine prediction error responses from reward to the next preceding conditioned stimulus without error backpropagation and thus without a ramp during the stimulus-reward interval (FIGURE 14, C AND D) (71,410,573). In particular, models using a biologically consistent striatal and midbrain circuit replicate well the dopamine risk ramp (581). Thus the dopamine ramp represents a genuine risk response and not the necessary byproduct of TD models.…”

Section: Popular Risk Measures Are Variancementioning

confidence: 79%

Neuronal Reward and Decision Signals: From Theories to Data

Schultz

2015

Physiological Reviews

923

765

View full text Add to dashboard Cite

LSchultz W. Neuronal Reward and Decision Signals: From Theories to Data. Physiol Rev 95: 853-951, 2015. Published June 24, 2015 doi:10.1152/physrev.00023.2014.-Rewards are crucial objects that induce learning, approach behavior, choices, and emotions. Whereas emotions are difficult to investigate in animals, the learning function is mediated by neuronal reward prediction error signals which implement basic constructs of reinforcement learning theory. These signals are found in dopamine neurons, which emit a global reward signal to striatum and frontal cortex, and in specific neurons in striatum, amygdala, and frontal cortex projecting to select neuronal populations. The approach and choice functions involve subjective value, which is objectively assessed by behavioral choices eliciting internal, subjective reward preferences. Utility is the formal mathematical characterization of subjective value and a prime decision variable in economic choice theory. It is coded as utility prediction error by phasic dopamine responses. Utility can incorporate various influences, including risk, delay, effort, and social interaction. Appropriate for formal decision mechanisms, rewards are coded as object value, action value, difference value, and chosen value by specific neurons. Although all reward, reinforcement, and decision variables are theoretical constructs, their neuronal signals constitute measurable physical implementations and as such confirm the validity of these concepts. The neuronal reward signals provide guidance for behavior while constraining the free will to act.

show abstract

Section: Popular Risk Measures Are Variancementioning

confidence: 79%

Neuronal Reward and Decision Signals: From Theories to Data

Schultz

2015

Physiological Reviews

923

765

View full text Add to dashboard Cite

show abstract

“…Recently, compelling evidence has been found indicating that the SNG may also be involved in the rewarding functions [42] or the responses related to reward-predicted stimuli [43]. For example, a study of a single-unit recording found that rats with high locomotors respond to novel context (i.e., more sensitized rats) and showed higher DA activations in the SNG relative to controls [44].…”

Section: Discussionmentioning

confidence: 99%

Neural Substrates of Amphetamine-Induced Behavioral Sensitization: Unconditioned (Zero Context) and Conditioned (Switch versus Same Context) Components in c-<b><i>fos</i></b> Overexpression

Wang

Yeh²,

Wang³

et al. 2012

Neuropsychobiology

View full text Add to dashboard Cite

The neural substrates of the unconditioned and conditioned components of amphetamine (AMPH)-induced behavioral sensitization remain unknown. The present study examines the brain activation of rats in response to an AMPH challenge with augmented locomotion in groups receiving chronic AMPH under chloral hydrate anesthetization (i.e., the ‘zero context’) or when tested in the ‘same context’ as a chronic treatment, or when tested in a ‘different context’. The neural activations of the three groups reveal fairly consistent patterns: (a) The substantia nigra is activated in the same context condition and the pure AMPH effect (i.e., the zero context with the unconditioned component), but not in the switch context condition. (b) The ventral pallidum showed Fos expression in the switch context and the same context, but not in the zero context condition. (c) The other nuclei, including the medial prefrontal cortex, nucleus accumbens, caudate putamen, medial thalamus, hippocampus, amygdala, and ventral tegmental area, are activated in all contextual conditions and the pure AMPH effect (the zero context). The context exerts definable effects on the mesocorticolimbic dopamine system on AMPH-induced behavioral sensitization. (d) The ventral pallidum and the substantia nigra activations dissociate the unconditioned component from the conditioned component in behavioral sensitization. Further studies are needed to determine how these two nuclei mediate the effect in terms of primary and conditioned rewards.

show abstract

“…Notable cases are: contingent cessation of an aversive input, contingent delivery ofnon-aversive stimuli that are novel but not linked to tangible rewards, and contingent access to the opportunity to engage in a more preferred behavior. Of these, the first two have been shown to have the effect on DA release that would be expected if it also mediates these components of internal reinforcement signaling (see also Ungless et al [8]), and some initial computational implications have been simulated with the help of the new striatal microcircuit model of Tan & Bullock [9]. The novelty-related DA cell responses present a further challenge to the classical view of DA signals as pure RPE signals.…”

Section: B Conditional Components Ofda Signaling Go Well Beyond Rpesmentioning

confidence: 99%

“…Under broad conditions, such co-release will produce a signal proportional to p(l-p), which shows a peak when uncertainty is maximal, i.e., whenp(R*ICS)=.5. Although the discoverers of this DA signal component aptly proposed that it may be important to explain habitual gambling, Tan & Bullock [9] noted that the broadcast of the DA signal to many brain sites beyond the dorsolateral striatum implies that it can function much more broadly, and adaptively, to optimize computations in both learning and performance. Notably, it can promote search for more-predictive representations, and rapid switching away from no-longer-rewarding alternatives.…”

Section: B Conditional Components Ofda Signaling Go Well Beyond Rpesmentioning

confidence: 99%

“…Another [19], as a function both of phasic and sustained dopamine signal inputs (shown in the lower panel) that reflect rewaard expectations, and ofthalamo-striatal glutarnatergic inputs (from CM-Pf nuclei, shown on the ordinate) that reflect stimulus salience. The lower panel shows the (normalized) learned DA response to a CS-then-reward sequence, after training with a p=0.5 conditional probability schedule during delay conditioning with a 2 second-long CS [9]. The phasic DA burst in response to CS onset (t=1.2 sec) reflects the expected value of reward; that in response to primary reward (t=3.2 sec) reflects the probabilistic schedule; and the gradual increase during the delay period reflects the uncertainty in the expected outcome [7;9;13].…”

Section: F Basal Ganglia Architecture Cannot Be Captured In Feedforwmentioning

confidence: 99%

See 1 more Smart Citation

Computational implications of microcircuit specializations in forebrain circuits for motivated action selection

Bullock

Tan

2009

2009 International Joint Conference on Neural Networks

View full text Add to dashboard Cite

Recent decades have seen dramatic progress in the brain sciences. Much attention has been attracted by non-invasive techniques, such as fMRI, which enable imaging of brain activities that support human cultural behavior, such as language. Less attention is paid to the cumulative progress in neural path-tracing and microcircuit specification in primates. The flood of such information poses a huge challenge for computational neuroscientists striving to bridge from microcircuits to flexible cognition, and to build models that can accurately predict individual differences in efficacy of neurological therapies. Recent models have made progress toward specifying how cortical circuits that enable planning and voluntary actions interact with adaptive sub-cortical microcircuits in the basal ganglia. The basal ganglia are key to understanding voluntary behavior, because they are strongly implicated in reinforcement learning and in all behavior over which the frontal lobes exert flexible control. This key role of the basal ganglia shows that ancient vertebrate designs have proven adaptable enough to support recent primate innovations, including speech and language generation. This paper summarizes recent models that have incorporated realistic representations of microcircuit features, and traced their com putational im plications. These efforts extend an emerging theoretical synthesis based on an interlocking set of com putational hypotheses regarding frontal cortex interactions with basal ganglia. These hypotheses specify how microcircuits utilize specialized electrical and chemical connections to solve learning and procedural control problems inherent to a massively parallel system.

show abstract

A Local Circuit Model of Learned Striatal and Dopamine Cell Responses under Probabilistic Schedules of Reward

Cited by 39 publications

References 81 publications

Neuronal Reward and Decision Signals: From Theories to Data

Neuronal Reward and Decision Signals: From Theories to Data

Neural Substrates of Amphetamine-Induced Behavioral Sensitization: Unconditioned (Zero Context) and Conditioned (Switch versus Same Context) Components in c-<b><i>fos</i></b> Overexpression

Computational implications of microcircuit specializations in forebrain circuits for motivated action selection

Contact Info

Product

Resources

About