Attenuated Directed Exploration during Reinforcement Learning in Gambling Disorder

Wiehler, Antonius; Chakroun, Karima; Peters, Jan

doi:10.1523/jneurosci.1607-20.2021

Cited by 39 publications

(101 citation statements)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This resembles the theoretically-predicted inverted-Ushaped relation of exploration, exploitation, and performance, as described e.g., by Addicott et al (2017). The disadvantages of diminished as well as excessive exploration can be observed in different psychiatric conditions: While several substance use disorders and gambling disorder are associated with diminished exploration (Morris et al, 2016;Wiehler et al, 2021), attention deficit hyperactivity syndrome and schizophrenia are associated with excessive exploration. The finding that increasing perseveration behavior, captured as 𝜌, leads to enhanced or unchanged performance in the bandit task contrasts with the conception of perseveration as a bounded rational strategy that saves cognitive resources on costs of performance accuracy (Gershman, 2020).…”

Section: Implications For Empirical Researchsupporting

confidence: 72%

“…The Kalman SMEP model accounted best for their data. We recently replicated this model comparison in a group of problem gambling participants and a group of healthy matched controls (Wiehler et al, 2021). Raja Beharelle et al (2015) found that the Kalman SME model accounted better than the Kalman SM model for a modified threearmed bandit task, which aims at preventing perseveration behavior.…”

Section: Introductionmentioning

confidence: 79%

“…The current study examined parameter and model recovery of reinforcement learning models for restless bandit problems. We focused on the Kalman SMEP model that has been shown to account for human data better than a range of competing (Chakroun et al, 2020;Wiehler et al, 2021), but we also examined restricted versions of that model, as well as Delta rule models, .94 -.96). Associations between true parameter values and model-free performance metrics confirmed that, for higher values of 𝛽, payout increased while switches decreased.…”

Section: Discussionmentioning

confidence: 99%

“…Even though the Kalman SMEP model was found to account best for bandit task data (Chakroun et al, 2020;Wiehler et al, 2021) parameter and model recovery work has been somewhat neglected, both by us and by others. Wilson and Collins (2019) suggest several steps researchers should follow to ensure the reliability and interpretability of computational modeling studies: first, an experiment or task has to be designed.…”

Section: Aim Of the Studymentioning

confidence: 93%

“…Research using model fitting to compare parameter values between populations or conditions (Addicott et al, 2021;Chakroun et al, 2020;Wiehler et al, 2021;Zajkowski et al, 2017) can regard the correlation of true and recovered parameter values as an indicator of statistical power. Small effects might not be detected, if the accuracy of parameter recovery is not sufficient (Wilson & Collins, 2019).…”

Section: Implications For Empirical Researchmentioning

confidence: 99%

See 4 more Smart Citations

Parameter and model recovery of reinforcement learning models for restless bandit problems

Danwitz

Mathar

Smith

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Multi-armed restless bandit tasks are regularly applied in psychology and cognitive neuroscience to assess exploration and exploitation behavior in structured environments. These models are also readily applied to examine effects of (virtual) brain lesions on performance, and to infer neurocomputational mechanisms using neuroimaging or pharmacological approaches. However, to infer individual, psychologically meaningful parameters from such data, computational cognitive modeling is typically applied. Recent studies indicate that softmax (SM) decision rule models that include a representation of environmental dynamics (e.g. the Kalman Filter) and additional parameters for modeling exploration and perseveration (Kalman SMEP) fit human bandit task data better than competing models. Parameter and model recovery are two central requirements for computational models: parameter recovery refers to the ability to recover true data-generating parameters; model recovery refers to the ability to correctly identify the true data generating model using model comparison techniques. Here we comprehensively examined parameter and model recovery of the Kalman SMEP model as well as nested model versions, i.e. models without the additional parameters, using simulation and Bayesian inference. Parameter recovery improved with increasing trial numbers, from around .8 for 100 trials to around .93 for 300 trials. Model recovery analyses likewise confirmed acceptable recovery of the Kalman SMEP model. Model recovery was lower for nested Kalman filter models as well as delta rule models with fixed learning rates.Exploratory analyses examined associations of model parameters with model-free performance metrics. Random exploration, captured by the inverse softmax temperature, was associated with lower accuracy and more switches. For the exploration bonus parameter modeling directed exploration, we confirmed an inverse-U-shaped association with accuracy, such that both an excess and a lack of directed exploration reduced accuracy. Taken together, these analyses underline that the Kalman SMEP model fulfills basic requirements of a cognitive model.

show abstract

Section: Implications For Empirical Researchsupporting

confidence: 72%

Section: Introductionmentioning

confidence: 79%

Section: Discussionmentioning

confidence: 99%

Section: Aim Of the Studymentioning

confidence: 93%

Section: Implications For Empirical Researchmentioning

confidence: 99%

See 3 more Smart Citations

Parameter and model recovery of reinforcement learning models for restless bandit problems

Danwitz

Mathar

Smith

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Parameter and Model Recovery of Reinforcement Learning Models for Restless Bandit Problems

et al. 2022

View full text Add to dashboard Cite

Multi-armed restless bandit tasks are regularly applied in psychology and cognitive neuroscience to assess exploration and exploitation behavior in structured environments. These models are also readily applied to examine effects of (virtual) brain lesions on performance, and to infer neurocomputational mechanisms using neuroimaging or pharmacological approaches. However, to infer individual, psychologically meaningful parameters from such data, computational cognitive modeling is typically applied. Recent studies indicate that softmax (SM) decision rule models that include a representation of environmental dynamics (e.g. the Kalman Filter) and additional parameters for modeling exploration and perseveration (Kalman SMEP) fit human bandit task data better than competing models. Parameter and model recovery are two central requirements for computational models: parameter recovery refers to the ability to recover true data-generating parameters; model recovery refers to the ability to correctly identify the true data generating model using model comparison techniques. Here we comprehensively examined parameter and model recovery of the Kalman SMEP model as well as nested model versions, i.e. models without the additional parameters, using simulation and Bayesian inference. Parameter recovery improved with increasing trial numbers, from around .8 for 100 trials to around .93 for 300 trials. Model recovery analyses likewise confirmed acceptable recovery of the Kalman SMEP model. Model recovery was lower for nested Kalman filter models as well as delta rule models with fixed learning rates. Exploratory analyses examined associations of model parameters with model-agnostic performance metrics. Random exploration, captured by the inverse softmax temperature, was associated with lower accuracy and more switches. For the exploration bonus parameter modeling directed exploration, we confirmed an inverse- U-shaped association with accuracy, such that both an excess and a lack of directed exploration reduced accuracy. Taken together, these analyses underline that the Kalman SMEP model fulfills two basic requirements of a cognitive model.

show abstract

Exploration–Exploitation Mechanisms in Recurrent Neural Networks and Human Learners in Restless Bandit Problems

Tuzsus,

Brands,

Pappas

et al. 2024

Comput Brain Behav

View full text Add to dashboard Cite

A key feature of animal and human decision-making is to balance the exploration of unknown options for information gain (directed exploration) versus selecting known options for immediate reward (exploitation), which is often examined using restless bandit tasks. Recurrent neural network models (RNNs) have recently gained traction in both human and systems neuroscience work on reinforcement learning, due to their ability to show meta-learning of task domains. Here we comprehensively compared the performance of a range of RNN architectures as well as human learners on restless four-armed bandit problems. The best-performing architecture (LSTM network with computation noise) exhibited human-level performance. Computational modeling of behavior first revealed that both human and RNN behavioral data contain signatures of higher-order perseveration, i.e., perseveration beyond the last trial, but this effect was more pronounced in RNNs. In contrast, human learners, but not RNNs, exhibited a positive effect of uncertainty on choice probability (directed exploration). RNN hidden unit dynamics revealed that exploratory choices were associated with a disruption of choice predictive signals during states of low state value, resembling a win-stay-loose-shift strategy, and resonating with previous single unit recording findings in monkey prefrontal cortex. Our results highlight both similarities and differences between exploration behavior as it emerges in meta-learning RNNs, and computational mechanisms identified in cognitive and systems neuroscience work.

show abstract

Attenuated Directed Exploration during Reinforcement Learning in Gambling Disorder

Cited by 39 publications

References 70 publications

Parameter and model recovery of reinforcement learning models for restless bandit problems

Parameter and model recovery of reinforcement learning models for restless bandit problems

Parameter and Model Recovery of Reinforcement Learning Models for Restless Bandit Problems

Exploration–Exploitation Mechanisms in Recurrent Neural Networks and Human Learners in Restless Bandit Problems

Contact Info

Product

Resources

About