Causal Model Comparison Shows That Human Representation Learning Is Not Bayesian

Abstract

How do we learn what features of our multidimensional environment are relevant in a given task? To study the computational process underlying this type of "representation learning," we propose a novel method of causal model comparison. Participants played a probabilistic learning task that required them to identify one relevant feature among several irrelevant ones. To compare between two models of this learning process, we ran each model alongside the participant during task performance, making predictions regarding the values underlying the participant's choices in real time. To test the validity of each model's predictions, we used the predicted values to try to perturb the participant's learning process: We crafted stimuli to either facilitate or hinder comparison between the most highly valued features. A model whose predictions coincide with the learned values in the participant's mind is expected to be effective in perturbing learning in this way, whereas a model whose predictions stray from the true learning process should not. Indeed, we show that in our task a reinforcement-learning model could help or hurt participants' learning, whereas a Bayesian ideal observer model could not. Beyond informing us about the notably suboptimal (but computationally more tractable) substrates of human representation learning, our manipulation suggests a sensitive method for model comparison, which allows us to change the course of people's learning in real time.We live in a rich, complex environment, in which we are constantly bombarded with a wide variety of sensory input. Even an action as simple as walking down the street carries with it a large volume of low-quality information in the form of people we see, places we walk by, cars, colors, voices, noises, emotional content, etc. Intuitively, one would imagine that given sufficient resources, it is best to always represent every aspect of the environment so that any detail can potentially be acted upon. However, the "curse of dimensionality" (Bellman 1957) posits that task representations that involve unnecessary stimulus dimensions will not afford efficient learning and decision-making, where efficiency is measured in the number of examples needed to learn the task. In particular, an increase in the number of dimensions of the problem (in our case, the dimensions of the environment that the brain may represent) implies that the learner needs to collect exponentially larger quantities of data to learn to solve the problem. If we want learning to be feasible it is therefore both computationally optimal and a practical imperative to represent tasks with as compact a representation as possible.What are the computational strategies that humans use to learn a representation for a given task? To address this question, we tested participants on a multidimensional trial-and-error choice task, in which only one dimension was relevant to predicting reward (Wilson and Niv 2012;Niv et al. 2015). To test the explanatory power of different models of learning dynamics, we develope...

Selective Maintenance of Value Information Helps Resolve the Exploration/Exploitation Dilemma

“…In this study of reinforcement-based timing a simple selective value maintenance strategy reduced information load and facilitated the transition to exploitation, further work is needed to evaluate alternative information maintenance strategies and their putative neural implementations. At the same time, our findings broadly align with emerging evidence that the implementation of optimal inference is limited by representational capacity (2) and that heuristic approaches to reinforcement learning are more likely to be effective in complex environments (20,57).…”

“…As we illustrate below, such selective maintenance trades off a high-fidelity representation of all available rewards for the opportunity to exploit the best response time (cf. 20).…”

Choice history effects in mice and humans improve reward harvesting efficiency

“…Evaluating whether animals follow optimal foraging principles faces several challenges. This is because, in experimental settings that approximate natural foraging, many variables such as energy, time, and opportunity costs are difficult to measure, and competing models can generate qualitatively similar predictions [10][11][12]. These problems can be mitigated in reward foraging tasks that require subjects to initiate decisions from identical starting points to equally distant options.…”

The effects of aging on the interaction between reinforcement learning and attention.

“…We have previously suggested that this process of “representation learning” depends on the interaction between RL and selective attention. Notably, attention filters what we learn about, and this attention filter is itself dynamically modulated by reinforcement (Geana & Niv, 2015; Niv et al, 2015; Wilson & Niv, 2012). Here we ask whether older adults differ from younger adults in their selective attention strategies and/or in the efficacy of their ability to learn from feedback in a multidimensional environment in which only some dimensions are relevant for the task at hand.…”