Combining error-driven models of associative learning with evidence accumulation models of decision-making

Abstract

As people learn a new skill, performance changes along two fundamental dimensions: Responses become progressively faster and more accurate. In cognitive psychology, these facets of improvement have typically been addressed by separate classes of theories. Reductions in response time (RT) have usually been addressed by theories of skill acquisition, whereas increases in accuracy have been explained by associative learning theories. To date, relatively little work has examined how changes in RT relate to changes in response accuracy, and whether these changes can be accounted for quantitatively within a single theoretical framework. The current work examines joint changes in accuracy and RT in a probabilistic category learning task. We report a model-based analysis of changes in the shapes of RT distributions for different category responses at the level of individual stimuli over the course of learning. We show that changes in performance are determined solely by changes in the quality of information entering the decision process. We then develop a new model that combines an associative learning front end with a sequential sampling model of the decision process, showing that the model provides a good account of all aspects of the learning data. We conclude by discussing potential extensions of the model and future directions for theoretical development that are opened up by our findings.

Prior Experience Modifies Acquisition Trajectories Via Response-Strategy Sampling

“…In principle, performance in 2AFC tasks can be described by popular associative or reinforcement learning theories (Daniels & Sanabria, 2018;Dayan & Daw, 2008;Kuchibhotla et al, 2019;Mill et al, 2014;Sewell et al, 2019). These models assume that in any 2AFC signal detection task, individuals seek to maximize outcomes by making choices with the greatest value and do so by learning the mapping and values between stimuli (tone plus white noise, white noise) responses (left lever, right lever), and outcomes (reward, no-reward).…”

A new model of decision processing in instrumental learning tasks

“…Recent advances (Fontanesi et al, 2019a, 2019b; Luzardo et al, 2017; McDougle and Collins, 2020; Miletić et al, 2020; Millner et al, 2018; Pedersen et al, 2017; Pedersen and Frank, 2020; Sewell et al, 2019; Sewell and Stallman, 2020; Shahar et al, 2019; Turner, 2019) have emphasized how both modelling traditions can be combined in joint models of reinforcement learning (RL) and evidence-accumulation decision-making processes, providing mutual benefits for both fields. Combined models generally propose that value-based decisionmaking and learning interact as follows: For each decision a subject gradually accumulates evidence for each choice option by sampling from a distribution of memory representations of the subjective value (or expected reward ) associated with each choice option (known as Q-values ).…”

“…The DDM is the dominant EAM as currently used in reinforcement learning(Fontanesi et al, 2019a, 2019b; Millner et al, 2018; Pedersen et al, 2017; Pedersen and Frank, 2020; Sewell et al, 2019; Sewell and Stallman, 2020; Shahar et al, 2019), but this choice is without experimental justification. Furthermore, the DDM has several theoretical drawbacks, such as its inability to explain multi-alternative decision-making and its strong commitment to the accumulation of the evidence difference , which leads to difficulties in explaining behavioral effects of absolute stimulus and reward magnitudes without additional mechanisms (Fontanesi et al, 2019a; Ratcliff et al, 2018; Teodorescu et al, 2016).…”

How spatial attention affects the decision process: looking through the lens of Bayesian hierarchical diffusion model & EEG analysis

“…There is a growing consensus that the evidence accumulates gradually and sequentially to make a decision ( 29,30 ). As a result, sequential sampling models (SSMs; Stone, 1960 ( 31 ), Ratcliff, Smith, Brown, and McKoon, 2016 ( 32 ) and Evans and Wagenmakers, 2020 ( 33 ) for the reviews), as the most well-known explanation of how the decision-making process works ( 26,30 ), have obtained very achievements in modeling the cognitive processes underlying decision making across a wide variety of paradigms, such as optimality polices ( [34][35][36][37] ), stop signal paradigms ( 38 ), go/no-go paradigms ( 39 ), multi-attribute & many alternatives choice ( [40][41][42] ), learning strategies ( [43][44][45] ), attentional choice ( [8][9][10]46 ), continuous responses ( 29,47 ), neural processes ( 1 ), and so on. In general, SSMs assume that decisions are made from a noisy process of accumulating evidence, that is to say, according to this theory, the evidence is gradually accumulated in favor of different choice alternatives over time with some rate until a sufficient amount of evidence for one of the options reaches a predetermined threshold to make a decision.…”