Reinforcement learning under uncertainty: expected versus unexpected uncertainty and state versus reward uncertainty

Abstract

Two prominent types of uncertainty that have been studied extensively are expected and unexpected uncertainty. Studies suggest that humans are capable of learning from reward under both expected and unexpected uncertainty when the source of variability is the reward. How do people learn when the source of uncertainty is the environment's state and rewards themselves are deterministic? How does their learning compare with the case of reward uncertainty? The present study addressed these questions using behavioural experimentation and computational modelling. Experiment 1 showed that human subjects were generally able to use reward feedback to successfully learn the task rules under state uncertainty, and were able to detect a non-signalled reversal of stimulus-response contingencies. Experiment 2, which combined all four types of uncertainties—expected versus unexpected uncertainty, and state versus reward uncertainty—highlighted key similarities and differences in learning between state and reward uncertainties. We found that subjects performed significantly better in the state uncertainty condition, primarily because they explored less and improved their state disambiguation. We also show that a simple reinforcement learning mechanism that ignores state uncertainty and updates the state-action value of only the identified state account for the behavioural data better than both a Bayesian reinforcement learning model that keeps track of belief states and a model that acts based on sampling from past experiences. Our findings suggest a common mechanism supports reward-based learning under state and reward uncertainty.

Error‐Correction Mechanisms in Language Learning: Modeling Individuals

“…(2018) previously provided evidence of disruption of classical conditioning performance when WM is loaded using dual‐task paradigms. The present finding adds to the mounting evidence that, against the predominant belief, WM may be implicated in low‐level cognitive processes such as instrumental learning, more commonly referred to as reinforcement learning within the neuroscience and machine learning communities (Collins & Frank, 2012; Ez‐zizi, 2016) and in some forms of implicit learning (Medimorec et al., 2021).…”