The drift diffusion model as the choice rule in reinforcement learning

Abstract

Current reinforcement-learning models often assume simplified decision processes that do not fully reflect the dynamic complexities of choice processes. Conversely, sequential-sampling models of decision making account for both choice accuracy and response time, but assume that decisions are based on static decision values. To combine these two computational models of decision making and learning, we implemented reinforcement-learning models in which the drift diffusion model describes the choice process, thereby capturing both within- and across-trial dynamics. To exemplify the utility of this approach, we quantitatively fit data from a common reinforcement-learning paradigm using hierarchical Bayesian parameter estimation, and compared model variants to determine whether they could capture the effects of stimulant medication in adult patients with attention-deficit hyper-activity disorder (ADHD). The model with the best relative fit provided a good description of the learning process, choices, and response times. A parameter recovery experiment showed that the hierarchical Bayesian modeling approach enabled accurate estimation of the model parameters. The model approach described here, using simultaneous estimation of reinforcement-learning and drift diffusion model parameters, shows promise for revealing new insights into the cognitive and neural mechanisms of learning and decision making, as well as the alteration of such processes in clinical groups.

The drift diffusion model as the choice rule in inter-temporal and risky choice: a case study in medial orbitofrontal cortex lesion patients and controls

“…In contrast, in perceptual decision-making, sequential sampling models such as the drift diffusion model (DDM) that not only account for the observed choices but also for the full reaction time distributions have a long tradition [8][9][10] . Recent work in reinforcement learning [11][12][13][14] intertemporal 15,16 and simple value-based choice [17][18][19][20] has shown that sequential sampling models can be successfully applied in these domains.…”

“…Recent studies on reinforcement learning have similarly used accuracy coding when fitting the DDM 13,14 to describe how choices and response time distributions relate to learned action values. In these studies, the upper boundary was defined as a selection of the stimulus with the objectively better reinforcement rate, and the lower boundary as a selection of the objectively inferior stimulus.…”

Attenuated directed exploration during reinforcement learning in gambling disorder

“…To examine group differences in the parameters of interest (β, φ, and ρ) we examined the posterior distributions of the group-level parameter means. Specifically, we report mean posterior group differences, standardized effect sizes for group differences and Bayes Factors testing for directional effects (Marsman & Wagenmakers, 2017;Pedersen, Frank, & Biele, 2017). Directional Bayes Factors (dBF) were computed as dBF = i / 1-i where i is the integral of the posterior distribution of the group difference from 0 to +∞, which we estimated via non-parametric density estimation.…”

Decomposing the effects of context valence and feedback information on speed and accuracy during reinforcement learning: A meta-analytical approach using diffusion decision modeling

“…However, most computational models of RL behavior only account for choices but neglect RTs. Therefore, how contextual modulations impact the relations between RTs and accuracy during learning has not been thoroughly explored so far (Summerfield and Tsetsos, 2012, but see Frank et al, 2015;Pedersen et al, 2017).…”