A Primer on Foraging and the Explore/Exploit Trade-Off for Psychiatry Research

Addicott, Merideth A.; Pearson, J. M.; Sweitzer, Maggie M.; Barack, David L.; Platt, Michael L.

doi:10.1038/npp.2017.108

Cited by 175 publications

(191 citation statements)

References 74 publications

(104 reference statements)

Supporting

Mentioning

184

Contrasting

Order By: Relevance

“…to try an alternative option with an uncertain but potentially higher payoff. This decision dilemma is commonly known as the "exploration/exploitation trade-off" (Cohen, McClure, & Yu, 2007;Addicott et al 2017). Striking a good balance between exploration and exploitation is essential for maximizing rewards and minimizing costs in the long term (Addicott et al, 2017).…”

Section: Introductionmentioning

confidence: 99%

“…This decision dilemma is commonly known as the "exploration/exploitation trade-off" (Cohen, McClure, & Yu, 2007;Addicott et al 2017). Striking a good balance between exploration and exploitation is essential for maximizing rewards and minimizing costs in the long term (Addicott et al, 2017). Too much exploitation prevents an agent from gathering new information in an uncertain environment, fosters inflexibility and habit formation.…”

Section: Introductionmentioning

confidence: 99%

“…Too much exploitation prevents an agent from gathering new information in an uncertain environment, fosters inflexibility and habit formation. Too much exploration, on the other hand, may lead to inefficient and inconsistent decision-making, thereby reducing long-term payoffs (Beeler 2014;Addicott 2017). Despite the high relevance of the explore/exploit trade-off for optimal decision-making, research is only beginning to unravel the mechanisms through which animals and humans solve this dilemma.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making

Chakroun

Mathar

Wiehler

et al. 2019

Preprint

View full text Add to dashboard Cite

Acknowledgements: This work was funded by Deutsche Forschungsgemeinschaft (grant PE1627/5-1 to J.P.). Large parts of this publication are based on the dissertation "Dopaminergic modulation of the explore/exploit trade-off in human decision making" by K.C. (Chakroun, Karima: Dopaminergic modulation of the explore/exploit trade-off in human decision making [online]; Hamburg, Univ., Diss., 2019; Summary A central issue in reinforcement learning and decision-making is whether to exploit knowledge of reward values, or to explore novel options. Although it is widely hypothesized that dopamine neurotransmission plays a key role in regulating this balance, causal evidence for a role of dopamine in human exploration is still lacking. Here, we use a combination of computational modeling, pharmacological intervention and functional magnetic resonance imaging (fMRI) to test for a causal effect of dopamine transmission on the exploration-exploitation trade-off in humans. 31 healthy male subjects performed a restless four-armed bandit task in a within-subjects design under three drug conditions: 150mg of the dopamine precursor L-dopa, 2mg of the D2 receptor antagonist haloperidol, and placebo. In all conditions, choice behavior was best explained by an extension of an established Bayesian learning model accounting for perseveration, uncertainty-based exploration and random exploration. Uncertainty-based exploration was attenuated under L-dopa compared to placebo and haloperidol. There was no evidence for a modulation of prediction error signaling or categorical effects of exploration/exploitation under L-dopa, whereas model-based fMRI revealed that L-dopa attenuated neural representations of overall uncertainty in insula and dorsal anterior cingulate cortex. Our results highlight the computational role of these regions in exploration and suggest that dopamine modulates exploration by modulating how this circuit tracks accumulating uncertainty during decision-making.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making

Chakroun

Mathar

Wiehler

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…While control dilemmas arise in a range of processing domains (e.g., goal shielding vs. goal shifting; focused attention vs. background-monitoring; anticipation of future needs vs. responding to current desires; computationally demanding but flexible goal-directed control vs. less demanding but inflexible habitual control, see below for a brief discussion), here we focus on the trade-off between exploration and exploitation as one of the most widely investigated control dilemmas (Blanchard and Gershman 2018;Cohen, McClure, and Yu 2007;Addicott et al 2017).…”

Section: Cognitive Control Dilemmasmentioning

confidence: 99%

Meta-control of the exploration-exploitation dilemma emerges from probabilistic inference over a hierarchy of time scales

Marković

Goschke

Kiebel

2019

Preprint

View full text Add to dashboard Cite

Cognitive control is typically understood as a set of mechanisms which enable humans to reach goals that require integrating the consequences of actions over longer time scales. Importantly, using routine beheavior or making choices beneficial only at a short time scales would prevent one from attaining these goals. During the past two decades, researchers have proposed various computational cognitive models that successfully account for behaviour related to cognitive control in a wide range of laboratory tasks. As humans operate in a dynamic and uncertain environment, making elaborate plans and integrating experience over multiple time scales is computationally expensive, the specific question of how uncertain consequences at different time scales are integrated into adaptive decisions remains poorly understood. Here, we propose that precisely the problem of integrating experience and forming elaborate plans over multiple time scales is a key component for better understanding how human agents solve cognitive control dilemmas such as the exploration-exploitation dilemma. In support of this conjecture, we present a computational model of probabilistic inference over hidden states and actions, which are represented as a hierarchy of time scales. Simulations of goal-reaching agents instantiating the model in an uncertain and dynamic task environment show how the exploration-exploitation dilemma may be solved by inferring meta-control states which adapt behaviour to changing contexts.

show abstract

“…The exploration-exploitation trade-off has been extensively studied in psychology and cognitive neurosciences, ranging from foraging studies in animals to human psychology and computational modeling studies (Cohen, McClure, & Yu, 2007;Daw, O'Doherty, Dayan, Seymour, & Dolan, 2006;Mehlhorn et al, 2015;Schulz & Gershman, 2019). A range of tasks has been developed to examine exploration in humans (see Addicott, Pearson, Sweitzer, Barack, & Platt (2017) for a review). One of the most widely used tasks is the multi-armed-bandit task, which also allows an examination of how exploration behavior unfolds over longer periods.…”

Section: Introductionmentioning

confidence: 99%

Attenuated directed exploration during reinforcement learning in gambling disorder

Wiehler

Chakroun

Peters

2019

Preprint

View full text Add to dashboard Cite

Gambling disorder is a behavioral addiction associated with impairments in decision-making and reduced behavioral flexibility. Decision-making in volatile environments requires a flexible trade-off between exploitation of options with high expected values and exploration of novel options to adapt to changing reward contingencies. This classical problem is known as the exploration-exploitation dilemma. We hypothesized gambling disorder to be associated with a specific reduction in directed (uncertainty-based) exploration compared to healthy controls, accompanied by changes in brain activity in a fronto-parietal explorationrelated network.Twenty-three frequent gamblers and nineteen matched controls performed a classical four-armed bandit task during functional magnetic resonance imaging. Computational modeling revealed that choice behavior in both groups contained signatures of directed exploration, random exploration and perseveration. Gamblers showed a specific reduction in directed exploration, while random exploration and perseveration were similar between groups.Neuroimaging revealed no evidence for group differences in neural representations of expected value and reward prediction errors. Likewise, our hypothesis of attenuated fronto-parietal exploration effects in gambling disorder was not supported. However, during directed exploration, gamblers showed reduced parietal and substantia nigra / ventral tegmental area activity. Cross-validated classification analyses revealed that connectivity in an exploration-related network was predictive of clinical status, suggesting alterations in network dynamics in gambling disorder.In sum, we show that reduced flexibility during reinforcement learning in volatile environments in gamblers is attributable to a reduction in directed exploration rather than an increase in perseveration. Neuroimaging findings suggest that patterns of network connectivity might be more diagnostic of gambling disorder than univariate value and prediction error effects. We provide a computational account of flexibility impairments in gamblers during reinforcement learning that might arise as a consequence of dopaminergic dysregulation in this disorder.

show abstract

A Primer on Foraging and the Explore/Exploit Trade-Off for Psychiatry Research

Cited by 175 publications

References 74 publications

Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making

Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making

Meta-control of the exploration-exploitation dilemma emerges from probabilistic inference over a hierarchy of time scales

Attenuated directed exploration during reinforcement learning in gambling disorder

Contact Info

Product

Resources

About