Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making

Abstract

Acknowledgements: This work was funded by Deutsche Forschungsgemeinschaft (grant PE1627/5-1 to J.P.). Large parts of this publication are based on the dissertation "Dopaminergic modulation of the explore/exploit trade-off in human decision making" by K.C. (Chakroun, Karima: Dopaminergic modulation of the explore/exploit trade-off in human decision making [online]; Hamburg, Univ., Diss., 2019; Summary A central issue in reinforcement learning and decision-making is whether to exploit knowledge of reward values, or to explore novel options. Although it is widely hypothesized that dopamine neurotransmission plays a key role in regulating this balance, causal evidence for a role of dopamine in human exploration is still lacking. Here, we use a combination of computational modeling, pharmacological intervention and functional magnetic resonance imaging (fMRI) to test for a causal effect of dopamine transmission on the exploration-exploitation trade-off in humans. 31 healthy male subjects performed a restless four-armed bandit task in a within-subjects design under three drug conditions: 150mg of the dopamine precursor L-dopa, 2mg of the D2 receptor antagonist haloperidol, and placebo. In all conditions, choice behavior was best explained by an extension of an established Bayesian learning model accounting for perseveration, uncertainty-based exploration and random exploration. Uncertainty-based exploration was attenuated under L-dopa compared to placebo and haloperidol. There was no evidence for a modulation of prediction error signaling or categorical effects of exploration/exploitation under L-dopa, whereas model-based fMRI revealed that L-dopa attenuated neural representations of overall uncertainty in insula and dorsal anterior cingulate cortex. Our results highlight the computational role of these regions in exploration and suggest that dopamine modulates exploration by modulating how this circuit tracks accumulating uncertainty during decision-making.

Intra-subject consistency of spontaneous eye blink rate in young women across the menstrual cycle

“…On the other hand, some PET studies in humans did not find a relation between EBR and DA synthesis capacity 31 or D2/3-receptor availability 30 . More recently, in a pharmacological study, Chakroun et al 58 found no modulation of the EBR by L-dopa (DA precursor) and haloperidol (D2 receptor antagonist).…”

“…After ovulation, the luteal phase is characterized by increasing levels of progesterone and medium estradiol levels 56 . However, some of the studies including the EBR as putative index for DA system only include males 5,29,57 , and others explicitly state that female sex hormones could affect their measurements 58 . This sex bias towards male subjects is even more pronounced in animal research ( 16,59 in monkeys 19 , in rodents).…”

Lapses in perceptual decisions reflect exploration

“…An intriguing candidate for this is dopamine, whose phasic responses have been shown to reflect state uncertainty (Starkweather et al, 2017;Babayan, Uchida, and Gershman, 2018;Lak et al, 2018), and whose tonic levels have been shown to modulate exploration in mice on a lever-press task (Beeler et al, 2010), and context-dependent song variability in songbirds (Leblois, Wendel, and Perkel, 2010). Dopaminergic genes have been shown to predict individual differences in uncertainty-guided exploration in humans (Frank et al, 2009), and dopaminergic disorders such as Parkinson's disease have been shown to disrupt the uncertainty-dependence of lapses across conditions on a multisensory task (Bertolini et al, 2015), while L-Dopa, a Parkinson's drug and dopamine precursor, has been shown to attentuate uncertainty-guided exploration (Chakroun et al, 2019). Patients with ADHD, another disorder associated with dopaminergic dysfunction, have been shown to display both increased perceptual variability and increased task-irrelevant motor output, a measure that correlates with lapses (Mihali et al, 2018).…”

Attenuated directed exploration during reinforcement learning in gambling disorder

“…This is particularly important in the context of clinical groups that are known to show increased perseveration (van Timmeren et al, 2018). We addressed this issue by extending existing models of exploration with an additional perseveration bonus term, such that final estimates of directed exploration are unconfounded by potential group differences in perseveration (Chakroun et al, 2019). Indeed, the full model including both directed exploration and perseveration terms accounted for the data best in both groups.…”

“…Traditionally, DA has been associated with reward prediction error coding and exploitation (Beeler, Daw, Frazier, & Zhuang, 2010;Pessiglione, Seymour, Flandin, Dolan, & Frith, 2006). But both theoretical accounts (Beeler, 2012) and empirical data (Chakroun, Mathar, Wiehler, Ganzer, & Peters, 2019;Cinotti et al, 2019;Frank, Doll, Oas-Terpstra, & Moreno, 2009;Gershman & Tzovaras, 2018;Kayser, Mitchell, Weinstein, & Frank, 2014) suggest a role of DA in regulating the explorationexploitation trade-off. DA also plays a role in problem gambling behavior.…”