From predictive models to cognitive models: Separable behavioral processes underlying reward learning in the rat

Abstract

Cognitive models are a fundamental tool in computational neuroscience, embodying in software precise hypotheses about the algorithms by which the brain gives rise to behavior. The development of such models is often largely a hypothesis-first process, drawing on inspiration from the literature and the creativity of the individual researcher to construct a model, and afterwards testing the model against experimental data. Here, we adopt a complementary data-first approach, in which richly characterizing and summarizing the patterns present in a dataset reveals an appropriate cognitive model, without recourse to an a priori hypothesis. We apply this approach to a large behavioral dataset from rats performing a dynamic reward learning task. The model revealed suggests that behavior on this task can be understood as a mixture of three components with different timescales: a quick-learning reward-seeking component, a slower-learning perseverative component, and a very slow "gambler's fallacy" component.

Mixing memory and desire: How memory reactivation supports deliberative decision-making.

“…Of interest is the fact that this combination of value learning and action selection function, though widely used, has been repeatedly shown to mis-estimate the shape of the choice probability curve in humans (Shteingart et al, 2013) and animals (Miller, Botvinick, et al, 2019) in the sorts of repeated decision-making tasks to which this family of models is best-suited -in particular, at the extremes. We return to this point below.…”

Mice exhibit stochastic and efficient action switching during probabilistic decision making

“…Lastly, as discussed above, both models require "stickiness" in action choice. This stickiness has been reported in analyses of behavior across tasks and species, and is also called perseveration, choice history bias, and the law of exercise (Thorndike 1911;Ito and Doya 2009;Balcarras et al 2016;Miller, Botvinick, and Brody 2018;Urai et al 2019;Lak et al 2020;Gershman 2020;Lai and Gershman 2021). We find that this bias to repeat previous actions offers a parsimonious mechanism for adapting an existing action policy to novel environmental conditions.…”

“…We next considered logistic regression, which has been used previously to describe rodent behavior in similar tasks (Tai et al 2012;Parker et al 2016;Donahue, Liu, and Kreitzer 2018;Miller, Botvinick, and Brody 2018), as an alternative model. Although this simpler model was shown to perform well at predicting the right and left choice of animals in these tasks, its ability to predict switches has not been evaluated.…”

“…The neural mechanisms underlying the decision-making strategy employed by animals in the multi-armed bandit are poorly understood, but are thought to involve basal ganglia and medial prefrontal cortex, including key inputs from neuromodulatory systems such as dopaminergic neurons (Ebitz, Albarran, and Moore 2018;Verharen, Adan, and Vanderschuren 2019;Bari et al 2019;Gershman and Uchida 2019). Recent work has shown the suitability of logistic regression and reinforcement learning-based models in predicting the choice behavior of agents, providing insight into how simple algorithms can reduce a series of actions and outcomes to features that are neurally tractable (Ito and Doya 2009;Miller, Botvinick, and Brody 2018). These models of behavior have facilitated the identification of neural correlates of action value representations, as well as neural activity corresponding to exploration in which the expressed behavior deviates from the action with the highest expected value (Tai et al 2012;Ebitz, Albarran, and Moore 2018;Donahue, Liu, and Kreitzer 2018;Verharen, Adan, and Vanderschuren 2019).…”

Limitations of a proposed correction for slow drifts in decision criterion

“…Previous studies have shown that a large subset of possible learning strategies can be approximated by logistic regression models that directly represent the influence of reward and choice history on future choices (Katahira 2015;Miller et al 2019). For this reason, in this study we explore the space of possible choice and outcome history effects instead of examining the correction's effect in the presence of individual learning algorithms.…”