Comparative Analysis of Behavioral Models for Adaptive Learning in Changing Environments

Abstract

Probabilistic models of decision making under various forms of uncertainty have been applied in recent years to numerous behavioral and model-based fMRI studies. These studies were highly successful in enabling a better understanding of behavior and delineating the functional properties of brain areas involved in decision making under uncertainty. However, as different studies considered different models of decision making under uncertainty, it is unclear which of these computational models provides the best account of the observed behavioral and neuroimaging data. This is an important issue, as not performing model comparison may tempt researchers to over-interpret results based on a single model. Here we describe how in practice one can compare different behavioral models and test the accuracy of model comparison and parameter estimation of Bayesian and maximum-likelihood based methods. We focus our analysis on two well-established hierarchical probabilistic models that aim at capturing the evolution of beliefs in changing environments: Hierarchical Gaussian Filters and Change Point Models. To our knowledge, these two, well-established models have never been compared on the same data. We demonstrate, using simulated behavioral experiments, that one can accurately disambiguate between these two models, and accurately infer free model parameters and hidden belief trajectories (e.g., posterior expectations, posterior uncertainties, and prediction errors) even when using noisy and highly correlated behavioral measurements. Importantly, we found several advantages of Bayesian inference and Bayesian model comparison compared to often-used Maximum-Likelihood schemes combined with the Bayesian Information Criterion. These results stress the relevance of Bayesian data analysis for model-based neuroimaging studies that investigate human decision making under uncertainty.

Adolescent risk-taking in the context of exploration and social influence

“…Eventually, the agents' behaviour will be optimally adapted to their experience with their environment (figure 1). Evidence suggests that learning to interact with novel environments can be characterized by some form of Bayesian learning (Daw et al, 2005(Daw et al, , 2006Dayan & Daw, 2008;Knill & Pouget, 2004;Marković & Kiebel, 2016;Mathys et al, 2011;Nassar et al, 2010). In Bayesian learning, in contrast to classic reinforcement learning, learning occurs faster when the learner experiences more uncertainty.…”

An empirical evaluation of active inference in multi-armed bandits

“…To derive the belief update equations we start with a hierarchical generative model described here and apply variational inference to obtain approximate learning rules. The obtained belief update equations correspond to the variational surprise minimisation learning (SMiLe) rule [45,46]. Importantly, we recover the learning rules for the stationary bandit (see 21) as a special case when changes are improbable.…”

“…Furthermore, learning algorithms derived from principles of Bayesian inference can be made domain-agnostic and fully adaptive to a wide range of unknown properties of the underlying bandit dynamics, such as the frequency of changes of choice-reward contingencies. Therefore, we use the same inference scheme for all algorithmsvariational surprise minimisation learning (SMiLE), an algorithm inspired by recent work in the field of human and animal decision making in changing environments [45,46]. The variational SMiLE algorithm corresponds to online Bayesian inference modulated by surprise, which can be expressed in terms of simple delta-like learning rules operating on the sufficient statistics of posterior beliefs.…”

Mind the Noise When Identifying Computational Models of Cognition from Brain Activity

“…Our idea of conducting a confusion analysis during the design phase of an experiment can be extended to address this issue, as suggested by the reviewer (see Text S1 in Devaine et al, 2014 or Marković and Kiebel, 2016 for examples). To that end, one would derive the full quadratic confusion matrices C ∈ ℝ M × M , with M denoting the number of models in the model space ${M}$.…”