Reinforcement Learning Under Uncertainty: Expected Versus Unexpected Uncertainty and State Versus Reward Uncertainty

Abstract

Two prominent types of uncertainty that have been studied extensively are expected and unexpected uncertainty. Studies suggest that humans are capable of learning from reward under both expected and unexpected uncertainty when the source of variability is the reward. How do people learn when the source of uncertainty is the environment’s state and the rewards themselves are deterministic? How does their learning compare with the case of reward uncertainty? The present study addressed these questions using behavioural experimentation and computational modelling. Experiment 1 showed that human subjects were generally able to use reward feedback to successfully learn the task rules under state uncertainty, and were able to detect a non-signalled reversal of stimulus-response contingencies. Experiment 2, which combined all four types of uncertainties—expected versus unexpected uncertainty, and state versus reward uncertainty—highlighted key similarities and differences in learning between state and reward uncertainties. We found that subjects performed significantly better in the state uncertainty condition, primarily because they explored less and improved their state disambiguation. We also show that a simple reinforcement learning mechanism that ignores state uncertainty and updates the state-action value of only the identified state accounted for the behavioural data better than both a Bayesian reinforcement learning model that keeps track of belief states and a model that acts based on sampling from past experiences. Our findings suggest a common mechanism supports reward-based learning under state and reward uncertainty.

Adaptive Integration of Perceptual and Reward Information in an Uncertain World

“…One key difference to our study is how perceptual uncertainty was induced. Whereas in our work and other previous studies (Bruckner et al, 2020; Lak et al, 2017; Lak et al, 2020; Sato & Kording, 2014; Vilares et al, 2012), perceptual information was associated with varying degrees of belief-state uncertainty, participants in Ez-zizi et al (2023) were presented with fixed stimuli calibrated to a pre-defined accuracy level. This potentially leaves little room and need for fine-tuning of learning.…”

“…This potentially leaves little room and need for fine-tuning of learning. Moreover, the computational model did not explicitly assume that reward probabilities changed throughout the task, potentially resulting in a worse model fit (Ez-zizi et al, 2023;Larsen et al, 2010).…”

“…This potentially leaves little room and need for fine-tuning of learning. Moreover, the computational model did not explicitly assume that reward probabilities changed throughout the task, potentially resulting in a worse model fit (Ez-zizi et al, 2023; Larsen et al, 2010). Future work could explicitly incorporate environmental changes into these models to further investigate the interplay of perceptual uncertainty and surprise (Bruckner et al, 2022).…”

“…Bayesian inference and reinforcement-learning approaches indicate that the degree of learning from new outcomes should be regulated to deal with perceptual uncertainty (Bruckner et al, 2020; Chrisman, 1992; Ez-zizi et al, 2023; Lak et al, 2017; Larsen et al, 2010). This dynamic regulation of learning behavior is typically quantified by the learning rate.…”

Decision-Making under Uncertainty

“…In contrast, when sensory information was deemed too uncertain, subjects ignored the stimulus altogether. Moreover, the study by Ez-Zizi et al (2023) combined perceptual uncertainty, risk, and environmental changes. Across two experiments, the task featured conditions combining perceptual uncertainty and environmental changes, as well as risky decision-making with changing reward contingencies.…”