A model for learning based on the joint estimation of stochasticity and volatility

Abstract

Influential research has stressed the importance of uncertainty for controlling the speed of learning, and of volatility (the inferred rate of change) in this process. This framework recasts biological learning as a problem of statistical inference, to produce prominent computational models that have extensive correlates in brain and behavior and burgeoning applications to psychopathology. Here, we investigate a neglected feature of these models, which is that learning rates are jointly determined by the comparison between volatility and a second factor, unpredictability (reflecting moment-to-moment stochasticity rather than systematic change). For the organism, volatility and unpredictability both manifest as noisy experiences, but for learning, they should have opposite effects: learning should speed up as volatility increases but slow down as unpredictability increases. Like volatility, unpredictability can vary and must be estimated by the learner. Previous work has focused on half this picture, studying how organisms estimate volatility while unpredictability is assumed fixed and known. The question how the brain estimates unpredictability (and ultimately, a full account of volatility, uncertainty and learning rates, which all depend on it) remains unaddressed. We introduce a new learning model, in which both unpredictability and volatility are learned from experience. We show evidence from the behavioral and decision neuroscience literatures that the brain distinguishes these two similar types of noise, so as to produce seemingly contradictory behavior in different situations. Further, the model highlights the extent to which inferences about volatility and unpredictability are interdependent, such that disruptions of computations related to volatility can impact computations related to unpredictability, and vice versa. We argue that this interdependence is apparent in learning following amygdala damage, and may have important implications for ongoing attempts to connect volatility and uncertainty to psychopathology.

What might interoceptive inference reveal about consciousness?

“…Interoceptive self-inference can be seen as a process theory built in part from the FEP, based on empirical and phenomenological observations [22,120]. In particular, the theory posits three core observations: I To persist, agents must learn to navigate a volatile, everchanging world [121][122][123].…”

Flexibility in valenced reinforcement learning computations across development

“…For example, in our task, all choices were equally likely to lead to positive and negative outcomes, and these reward probabilities were perfectly stable for the duration of learning. Few choices in the 'real world' are equally likely to lead to good and bad outcomes, and often, individuals face changing environments where they must dissociate the stochasticity and volatility of reward outcomes (Nassar et al, 2010;Piray & Daw, 2020). Thus, the general learning biases we observed may be specific to the design of our task (Eckstein, Master, Xia, et al, 2021).…”

“…In other words, while individuals' positive and negative learning rates determined the extent to which they weighted recent positive and negative prediction errors in updating their beliefs about the reward structure of the environment, their beliefs in turn likely shaped how they learned from experienced outcomes. At the computational level, there are multiple plausible mechanisms for how individuals may tune valenced learning rates to different environments -individuals may track the volatility and stochasticity of reward outcomes and use the variability of prediction errors or rates at which prediction errors change to scale learning rates (Behrens et al, 2007;Cazé & van der Meer, 2013;Diederen & Schultz, 2015;Gershman, 2015;McGuire et al, 2014;Nassar et al, 2010Nassar et al, , 2012Piray & Daw, 2020). Models that dynamically adjust learning rates based on experienced outcomes could also yield further insight into the block order effects we observed by allowing for learned information about the environment's reward statistics to be carried over into new contexts.…”

Trait Somatic Anxiety is Associated With Reduced Directed Exploration and Underestimation of Uncertainty

“…We do this by varying volatility (Piray & Daw, 2020), that is, the change rate of an option's true value (e.g., the update speed of the pastry menus). Though the uncertainty-driven exploration framework predicts that the impact of uncertainty on exploration is independent of the way it is varied, this hypothesis is yet to be tested.…”