A new model of decision processing in instrumental learning tasks

Miletić, Steven; Boag, Russell J.; Trutti, Anne C.; Forstmann, Birte U.; Heathcote, Andrew

doi:10.1101/2020.09.12.294512

Cited by 9 publications

(17 citation statements)

References 101 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DDMs fall under the umbrella of evidence accumulation models which describe the process of action selection and resulting reaction times as a biased random walk with a drift and white noise. This approach has been extended to multi-choice tasks in so-called race diffusion models, where instead of having one accumulator as in the DDM, each available choice option is associated with a different accumulator ( Fontanesi et al, 2019 ; Miletić et al, 2021 ).…”

Section: Discussionmentioning

confidence: 99%

“…The evidence accumulation reaction time modeling approach has been recently combined with reinforcement learning models to provide joint instrumental learning and reaction time models ( Milosavljevic et al, 2010 ; Pedersen et al, 2017 ; Fontanesi et al, 2019 ; Miletić et al, 2021 ). Here, usually internal variables from the reinforcement learning agent, particularly expected reward (Q-values) are mapped to variables in the evidence accumulator model, such as the drift rate, e.g.…”

Section: Discussionmentioning

confidence: 99%

“…Recently, there have been some successful joint applications of value-based decision models, specifically reinforcement learning, and reaction time models ( Milosavljevic et al, 2010 ; Pedersen et al, 2017 ; Fontanesi et al, 2019 ; Shahar et al, 2019 ; Miletić et al, 2021 ), which are typically evidence accumulator models such as diffusion decision models (DDM) ( Ratcliff, 1978 ; Forstmann and Wagenmakers, 2015 ) and so-called race diffusion models ( Milosavljevic et al, 2010 ). Here, the principled idea is to connect choice values and probabilities to reaction times by linking parameters in both models.…”

Section: Introductionmentioning

confidence: 99%

“…For example, the trial-wise expected reward (Q-values) in reinforcement learning models can be used to vary the drift rate of a DDM ( Pedersen et al, 2017 ). This approach has recently been extended to multi-choice tasks using race diffusion models, where instead of having one accumulator as in the DDM, each available choice option is associated with a different accumulator ( Fontanesi et al, 2019 ; Miletić et al, 2021 ).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Joint modeling of choices and reaction times based on Bayesian contextual behavioral control

Schwöbel

Marković

Smolka

et al. 2021

Preprint

View full text Add to dashboard Cite

In cognitive neuroscience and psychology, reaction times are an important behavioral measure. However, in instrumental learning and goal-directed decision making experiments, findings often rely only on choice probabilities from a value-based model, instead of reaction times. Recent advancements have shown that it is possible to connect value-based decision models with reaction time models, for example in a joint reinforcement learning and diffusion decision model. We propose a novel joint model of both choices and reaction times by combining a mechanistic account of Bayesian sequential decision making with a sampling procedure. Specifically, we use a recent context-specific Bayesian forward planning model which we extend by an MCMC sampler to obtain both choices and reaction times. We show that we can explain and reproduce well-known experimental findings in value based-decision making as well as classical inhibition and switching tasks. First, we use the proposed model to explain how instrumental learning and automatized behaviour result in decreased reaction times and improved accuracy. Second, we reproduce classical results in the Eriksen flanker task. Third, we reproduce established findings in task switching. These findings show that the proposed joint behavioral model may describe common underlying processes in all these types of decision making paradigms.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Joint modeling of choices and reaction times based on Bayesian contextual behavioral control

Schwöbel

Marković

Smolka

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The evidence from successive samples is summed, or accumulated through time until a criterion amount of evidence for one response alternative is accrued, initiating a behavioral response for that alternative. The success of this theoretical framework is reflected in the breadth of domains the models have been applied to, such as evaluating the optimality of decision policies (Bogacz et al, 2006;Drugowitsch et al, 2012;Evans and Brown, 2017;Evans et al, 2018;Starns and Ratcliff, 2012), stop signal paradigms (Matzke et al, 2013(Matzke et al, , 2017a, Go/No-Go paradigms (Gomez et al, 2007;Ratcliff et al, 2018), multi-attribute and many-alternatives choice Diederich, 2019, 2021;Kvam, 2019;Roe et al, 2001;Trueblood et al, 2014;Usher and McClelland, 2004), learning strategies (Fontanesi et al, 2019;Miletić et al, 2021;Pedersen et al, 2017;Sewell et al, 2019;Sewell and Stallman, 2020), attentional choice (Krajbich et al, 2010(Krajbich et al, , 2012Gluth et al, 2020), continuous responses (Ratcliff, 2018;Smith, 2016), neural processes (Gold and Shadlen, 2007), and so on.…”

Section: Introductionmentioning

confidence: 99%

Are there jumps in evidence accumulation, and what, if anything, do they reflect psychologically? An analysis of Lévy-Flights models of decision-making

Rasanan¹,

Ja²,

Dk³

2021

Preprint

View full text Add to dashboard Cite

According to existing theories of simple decision-making, decisions are initiated by continuously sampling and accumulating perceptual evidence until a threshold value has been reached. Many models, such as the diffusion decision model, assume a noisy accumulation process, described mathematically as a stochastic Wiener process with Gaussian distributed noise. Recently, an alternative account of decision-making has been proposed in the Lévy Flights (LF) model, in which accumulation noise is characterized by a heavy-tailed power-law distribution, controlled by a parameter, α. The LF model produces sudden large “jumps” in evidence ac- cumulation that are not produced by the standard Wiener diffusion model, which some have argued provide better fits to data. It remains unclear, however, whether jumps in evidence accumulation have any real psychological meaning. Here, we investigate the conjecture by Voss et al. (2019) that jumps might reflect sudden shifts in the source of evidence people rely on to make decisions. We reason that if jumps are psychologically real, we should observe systematic reductions in jumps as people become more practiced with a task (i.e., as people converge on a stable decision strategy with experience). We fitted four versions of the LF model to behavioral data from a study by Evans and Brown (2017), using a five-layer deep inference neural network for parameter estimation. The analysis revealed systematic reductions in jumps as a function of practice, such that the LF model more closely approximated the standard Wiener model over time. This trend could not be attributed to other sources of parameter variability, speaking against the possibility of trade-offs with other model parameters. Our analysis suggests that jumps in the LF model might be capturing strategy instability exhibited by relatively inexpe- rienced observers early on in task performance. We conclude that further investigation of a potential psychological interpretation of jumps in evidence accumulation is warranted.

show abstract

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

Colas

Dundon

Gerraty

et al. 2022

Human Brain Mapping

View full text Add to dashboard Cite

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single rewardprediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversallearning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With highresolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

show abstract

A new model of decision processing in instrumental learning tasks

Cited by 9 publications

References 101 publications

Joint modeling of choices and reaction times based on Bayesian contextual behavioral control

Joint modeling of choices and reaction times based on Bayesian contextual behavioral control

Are there jumps in evidence accumulation, and what, if anything, do they reflect psychologically? An analysis of Lévy-Flights models of decision-making

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

Contact Info

Product

Resources

About