Learning in Volatile Environments With the Bayes Factor Surprise

Abstract

Surprise-based learning allows agents to rapidly adapt to nonstationary stochastic environments characterized by sudden changes. We show that exact Bayesian inference in a hierarchical model gives rise to a surprise-modulated trade-off between forgetting old observations and integrating them with the new ones. The modulation depends on a probability ratio, which we call the Bayes Factor Surprise, that tests the prior belief against the current belief. We demonstrate that in several existing approximate algorithms, the Bayes Factor Surprise modulates the rate of adaptation to new observations. We derive three novel surprise-based algorithms, one in the family of particle filters, one in the family of variational learning, and one in the family of message passing, that have constant scaling in observation sequence length and particularly simple update dynamics for any distribution in the exponential family. Empirical results show that these surprise-based algorithms estimate parameters better than alternative approximate approaches and reach levels of performance comparable to computationally more expensive algorithms. The Bayes Factor Surprise is related to but different from the Shannon Surprise. In two hypothetical experiments, we make testable predictions for physiological indicators that dissociate the Bayes Factor Surprise from the Shannon Surprise. The theoretical insight of casting various approaches as surprise-based learning, as well as the proposed online algorithms, may be applied to the analysis of animal and human behavior and to reinforcement learning in nonstationary environments.

Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

“…To adapt both model-based and model-free policies of the SurNoR algorithm, surprise is used in two different ways. First, high values of surprise systematically lead to a larger learning rate for the update of the world-model than smaller ones, consistent with earlier models [27,29]. Second, going beyond previous models of behavior [20,[24][25][26]30], surprise also influences the learning rate of the model-free reinforcement learning branch.…”

“…Our key findings can be summarized in three points: (i) We find that novelty-seeking explains participants' exploratory behavior better than alternative exploration strategies such as seeking surprise or uncertainty [42,43]; (ii) we observe that participants use their worldmodel only rarely for action planning and mainly to extract moments of surprise; and importantly, (iii) we show that surprise calculated by the world-model does not only modulate the learning of the world-model [24][25][26]29] but also the learning of model-free action-values. In particular, we show that such a modulation is necessary to explain participants' adaptive behavior.…”

“…To quantify expectations, we assume that participants build an internal model of the environment ('world-model'), i.e., we hypothesize that participants estimate the probability p (t) (s t+1 |s t , a t ) of a transition from a given state s t to another state s t+1 when performing action a t . More precisely, we assume that the world-model counts transitions from state s to s 0 under action a using either a leaky [23,49,50] or a surprise-modulated [28,29,51] counting procedure, described by the pseudo-countC ðtÞ s;a!s 0 . The conditional probability is then p ðtÞ ðs tþ1 js t ; a t Þ ¼C ðtÞ s t ;a t !s tþ1 þ � C ðtÞ s t ;a t þ 11�…”

“…Therefore, we consider the surprise of such a transition to be a decreasing function of p (t) (s t+1 |s t , a t ). More precisely, we use a recent measure of surprise motivated by a Bayesian framework for learning in volatile environments, called the 'Bayes Factor' surprise [29]. The Bayes Factor surprise of the transition from state s t to state s t+1 after taking action a t is S ðtþ1Þ BF ¼ const: p ðtÞ ðs tþ1 js t ; a t Þ ; ð4Þ…”

“…Whereas the reward prediction error (RPE) is a mismatch between the expected reward and the actual reward, surprise is a mismatch between an expected observation and an actual observation. Behavioral experiments [20,[24][25][26][27] and theories [27][28][29][30] suggest that surprise helps humans to adapt their behavior quickly to changes in the environment, potentially by modulating synaptic plasticity [31][32][33].…”