Striatal structure and function predict individual biases in learning to avoid pain

Eldar, Eran; Hauser, Tobias U.; Dayan, Peter; Dolan, Raymond J.

doi:10.1073/pnas.1519829113

Cited by 70 publications

(77 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To this end, we used the winning computational model and extracted the model predictions for each trial. To derive the model predictions, we used the average parameter estimates across all subjects, similar to previous studies (43,(80)(81)(82)(83)(84). This ensures more regularized predictions and does not introduce subject-specific biases.…”

Section: Methodsmentioning

confidence: 99%

“…We used the total scores as a dependent variable in a fivefold crossvalidated regression (cf. 84,97). To assess whether apathy was more closely linked to dmPFC or VS activation, we used the activation in the given ROI (using mean activation at P < 0.05 FWE ROI, same as in previous analyses), including both effort and reward prediction error signals at the time of outcome.…”

Section: Methodsmentioning

confidence: 99%

“…This means that subjects with a strong expression of a dmPFC gray matter density should also express greater gray matter density in ventrolateral SN/VTA, possibly reflecting genetic, developmental, or environmental influences (59). We used the segmented, normalized gray matter MT maps and applied the Jacobian of the normalization step to preserve total tissue volume (103), as reported in a previous study (84). To account for differences in global brain volume, we calculated the total intracranial volume and used it as a nuisance regressor in the analysis.…”

Section: Methodsmentioning

confidence: 99%

See 2 more Smart Citations

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

Hauser

Eldar

Dolan

2017

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

122

127

View full text Add to dashboard Cite

Optimal decision making mandates organisms learn the relevant features of choice options. Likewise, knowing how much effort we should expend can assume paramount importance. A mesolimbic network supports reward learning, but it is unclear whether other choice features, such as effort learning, rely on this same network. Using computational fMRI, we show parallel encoding of effort and reward prediction errors (PEs) within distinct brain regions, with effort PEs expressed in dorsomedial prefrontal cortex and reward PEs in ventral striatum. We show a common mesencephalic origin for these signals evident in overlapping, but spatially dissociable, dopaminergic midbrain regions expressing both types of PE. During action anticipation, reward and effort expectations were integrated in ventral striatum, consistent with a computation of an overall net benefit of a stimulus. Thus, we show that motivationally relevant stimulus features are learned in parallel dopaminergic pathways, with formation of an integrated utility signal at choice. effort prediction errors | reward prediction errors | apathy | substantia nigra/ventral tegmental area | dorsomedial prefrontal cortex O rganisms need to make energy-efficient decisions to maximize benefits and minimize costs, a tradeoff exemplified in effort expenditure (1-3). A key example occurs during foraging, where an overestimation of effort can lead to inaction and starvation (4), whereas underestimation of effort can result in persistent failure, as exemplified in the myth of Sisyphus (5).In a naturalistic environment, we often simultaneously learn about success in expending sufficient effort into an action as well as the reward we obtain from this same action. The reward outcomes that signal success and failure of an action are usually clear, although the effort necessary to attain success is often less transparent. Only by repeatedly experiencing success and failure is it possible to acquire an estimate of an optimal level of effort needed to succeed, without unnecessary waste of energy. This type of learning is important in contexts as diverse as foraging, hunting, and harvesting (6-8). Hull in his "law of less work" proposed that organisms "gradually learn" how to minimize effort expenditure (9). Surprisingly, we know little regarding the neurocognitive mechanisms that guide this form of simultaneous learning about reward and effort.A mesolimbic dopamine system encodes a teaching signal tethered to prediction of reward outcomes (10, 11). These reward prediction errors (PEs) arise from dopaminergic neurons in substantia nigra and ventral tegmental area (SN/VTA) and are broadcast to ventral striatum (VS) to mediate reward-related adaptation and learning (12, 13). Dopamine is also thought to provide a motivational signal (14-18), while dopaminergic deficits in rodents impair how effort and reward are arbitrated (1, 4, 19). The dorsomedial prefrontal cortex (dmPFC; spanning presupplementary motor area [pre-SMA] and dorsal anterior cingulate cortex [dACC]) is a candidate substrate...

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

Hauser

Eldar

Dolan

2017

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

122

127

View full text Add to dashboard Cite

show abstract

“…For pain, this involves escape (from persistent pain) and avoidance (of phasic pain). Current evidence suggests that the brain employs a parallel system in which action values for relief and pain are simultaneously learned and interact to guide choice (Seymour et al, 2012;Eldar et al, 2016). Indeed, there is a specific advantage to learning these two values separately: relief values can approximate the bestcase scenario of future actions (''what to do''), and pain values can approximate the worst-case scenarios (''what not to do''), and the two values can in principle be integrated together to guide action (''multi-attribute RL'').…”

Section: Instrumental Learningmentioning

confidence: 99%

Pain: A Precision Signal for Reinforcement Learning and Control

Seymour

2019

Neuron

115

104

View full text Add to dashboard Cite

Since noxious stimulation usually leads to the perception of pain, pain has traditionally been considered sensory nociception. But its variability and sensitivity to a broad array of cognitive and motivational factors have meant it is commonly viewed as inherently imprecise and intangibly subjective. However, the core function of pain is motivational-to direct both short-and long-term behavior away from harm. Here, we illustrate that a reinforcement learning model of pain offers a mechanistic understanding of how the brain supports this, illustrating the underlying computational architecture of the pain system. Importantly, it explains why pain is tuned by multiple factors and necessarily supported by a distributed network of brain regions, recasting pain as a precise and objectifiable control signal.Despite the advent of brain imaging, a clear picture of how pain is processed in the brain has been much harder to unravel than anticipated, being beset by three problems. First, pain is associated with robust responses in multiple and diverse brain regions, most of which are not specific to pain (at least on a macroscopic scale), and so it has been hard to ''pin down'' the pain system to any specific brain region. Second, pain is an inherently private percept, but an individual's self-reports of pain can vary widely from moment to moment, and it has remained unclear whether this fluctuation represents irreducible noise and subjectivity or a precise tuning of pain based on hidden factors. Third, pain is exquisitely sensitive to a broad range of emotional, environmental, and cognitive factors-a phenomenon called endogenous modulation. Although this has led to an appreciation that pain is more than a simple readout of nociceptive input, it has not led to any satisfactory unified explanation as to what pain really is. This has left the view that pain is simply a highly variable and malleable representation of assumed actual or potential tissue damage.In this review, we propose a model of pain that centralizes its role as a learning and control signal and argue that this can solve these problems. We begin with a perspective of how theories of pain have evolved over recent decades, and how insights have emerged that have moved thinking beyond purely sensory accounts of pain. We then argue that current accounts still don't fully capture how pain controls behavior to minimize harm, which is its primary function. Importantly, although this is often achieved by immediate nocifensive responses, a substantial part of this comes from learning-allowing an animal to mitigate or avoid predictable harm long into the future. The foundations of a learning account of pain are rooted in psychological models of animal learning, and we describe how these can be developed in computational terms to provide a mechanistic model of the architecture of the pain system. Critically, we argue that this requires pain to be shaped by a set of factors to optimize its role as a learning and control signal and review evidence that suggests that a great...

show abstract

“…The anterior insula and striatum (more ventrally) have previously been implicated in representing expected value and prediction error signals in higher-order pain conditioning (Seymour et al, 2004), and the dorsal striatum is implicated in prediction error signals in avoidance learning (Palminteri et al, 2012; Seymour et al, 2012; Eldar et al, 2016), suggesting an important role for these structures in aversive learning (see also Delgado et al, 2009). Dorsal, rather than more ventral striatal control has also been implicated in the transfer from goal-directed to habit-based avoidance in instrumental paradigms (LeDoux et al, 2017).…”

Section: Discussionmentioning

confidence: 99%

Value generalization in human avoidance learning

Norbury

Robbins

Seymour

2017

Preprint

View full text Add to dashboard Cite

SummaryGeneralization during aversive decision-making allows us to avoid a broad range of potential threats following experience with a limited set of exemplars. However, over-generalization, resulting in excessive and inappropriate avoidance, has been implicated in a variety of psychological disorders. Here, we use reinforcement learning modelling to dissect out different contributions to the generalization of instrumental avoidance in two groups of human volunteers (N=26, N=482). We found that generalization of avoidance could be parsed into perceptual and value-based processes, and further, that value-based generalization could be subdivided into that relating to aversive and neutral feedback - with corresponding circuits including primary sensory cortex, anterior insula, and ventromedial prefrontal cortex, respectively. Further, generalization from aversive, but not neutral, feedback was associated with self-reported anxiety and intrusive thoughts. These results reveal a set of distinct mechanisms that mediate generalization in avoidance learning, and show how specific individual differences within them can yield anxiety.

show abstract

Striatal structure and function predict individual biases in learning to avoid pain

Cited by 70 publications

References 52 publications

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

Pain: A Precision Signal for Reinforcement Learning and Control

Value generalization in human avoidance learning

Contact Info

Product

Resources

About