A simple model for learning in volatile environments

Abstract

Sound principles of statistical inference dictate that uncertainty shapes learning. In this work, we revisit the question of learning in volatile environments, in which both the first and second-order statistics of observations dynamically evolve over time. We propose a new model, the volatile Kalman filter (VKF), which is based on a tractable state-space model of uncertainty and extends the Kalman filter algorithm to volatile environments. The proposed model is algorithmically simple and encompasses the Kalman filter as a special case. Specifically, in addition to the error-correcting rule of Kalman filter for learning observations, the VKF learns volatility according to a second error-correcting rule. These dual updates echo and contextualize classical psychological models of learning, in particular hybrid accounts of Pearce-Hall and Rescorla-Wagner. At the computational level, compared with existing models, the VKF gives up some flexibility in the generative model to enable a more faithful approximation to exact inference. When fit to empirical data, the VKF is better behaved than alternatives and better captures human choice data in two independent datasets of probabilistic learning tasks. The proposed model provides a coherent account of learning in stable or volatile environments and has implications for decision neuroscience research.

Paranoia as a deficit in non-social belief updating

“…Therefore, a more complicated model, one that captures higher-level beliefs about contingency transitions or learning when to learn, seems most appropriate, and indeed, that type of model was able to simulate the key features of our data ( Palminteri et al, 2017 ). Future work will compare and contrast different potential computational models included, but not limited to Bayesian Hidden State Markov Models ( Hampton et al, 2006 ), as well as switching ( Gershman et al, 2014 ) and volatile Kalman Filters ( Piray and Daw, 2020 ).…”

A model for learning based on the joint estimation of stochasticity and volatility

“…But the very premise of these experiments violates the simplifying assumption of the Kalman filter -that volatility is fixed and known to the agent. To handle this situation, new models were developed (Mathys et al, 2011;Piray and Daw, 2020) that generalize the Kalman filter to incorporate learning the volatility " as well, arising from Bayesian inference in a hierarchical generative model in which the true " is also changing. In this case, exact inference is no longer tractable, but approximate inference is possible and typically incorporates Eqs.…”

“…We developed a probabilistic model for learning under these circumstances. The data generation process arises from a further hierarchical generalization of these models (especially the generative model used in our recent work (Piray and Daw, 2020)), in which the true value of unpredictability " is unknown and changing, as are the true reward rate and volatility ( Figure 1d). The goal of the learner is to estimate the true reward rate from observations, which necessitates inferring volatility and unpredictability as well.…”

“…First consider volatility. It has been argued that the Pearce-Hall (1980) logic is formalized by volatilitylearning models (Courville et al, 2006;Dayan et al, 2000;Piray and Daw, 2020). In these models, surprising outcomes during pretraining increase inferred volatility and thus speed subsequent relearning.…”

“…The connection between uncertainty and learning rate has inspired an influential series of hierarchical Bayesian inference models, elaborating a baseline model known as the Kalman filter (Behrens et al, 2007;Mathys et al, 2011;Piray and Daw, 2020). A key feature of these models is that learning occurs over multiple trials, so the updated posterior at one step becomes the prior at the next -with the important additional twist that uncertainty increases at each step due to the possibility that the true value has changed.…”