A simple model for learning in volatile environments

Abstract

Sound principles of statistical inference dictate that uncertainty shapes learning. In this work, we revisit the question of learning in volatile environments, in which both the first and second-order statistics of environments dynamically evolve over time. We propose a new model, the volatile Kalman filter (VKF), which is based on a tractable state-space model of uncertainty and extends the Kalman filter algorithm to volatile environments. Algorithmically, the proposed model is simpler and more transparent than existing models, and encompasses the Kalman filter as a special case. Specifically, in addition to the errorcorrecting rule of Kalman filter for learning observations, the VKF learns volatility according to a second error-correcting rule. These dual updates echo and contextualize classical psychological models of learning, in particular hybrid accounts of Pearce-Hall and Rescorla-Wagner. At the computational level, compared with existing models, the VKF is more accurate, particularly in estimating volatility, as it is based on more faithful approximations to the exact inference. Accordingly, when fit to empirical data, the VKF is better behaved than alternatives and better captures human choice data in a probabilistic learning task.The proposed model provides a transparent and coherent account of learning in stable or volatile environments and has implications for decision neuroscience research.

Impairments in Probabilistic Prediction and Bayesian Learning Can Explain Reduced Neural Semantic Priming in Schizophrenia

“…Sharpe et al prior beliefs about environmental instability. 68,69 What the present set of findings do suggest is that patients were unable to distinguish between inputs that were unexpected because of their uncertainty about the statistical structure of the current environment (expected uncertainty) and inputs that were unexpected because of a true change in the environment (unexpected uncertainty). 70 These findings, therefore, extend previous proposals that impairments in inferring the precision of prediction error lead to abnormalities in perception and belief in schizophrenia 6,71 by raising the possibility that the same computational impairments might underlie impairments in learning.…”

A Computational Role for Top–Down Modulation from Frontal Cortex in Infancy

“…The latter regressor arises in a family of elaborations of this model, which have been used in a number of different applications in behavioral neuroscience. These posit a role for the unsigned error magnitude (often measured by δ 2 ) in tracking the volatility of the environment in a hierarchical model and using it, top-down, to modulate the rate of learning predictions (Piray & Daw, 2019;Li et al, 2011;Behrens, Woolrich, Walton, & Rushworth, 2007;Pearce & Hall, 1980). We used the model parameters that were reported previously for a hybrid of RW and Pearce-Hall (Pearce & Hall, 1980) models (Li et al, 2011) to generate the trial-by-trial prediction error and its magnitude for each infant according to the specific sequence of trials administered.…”