Belief states and categorical-choice biases determine reward-based learning under perceptual uncertainty

Abstract

In natural settings, learning and decision making often takes place under considerable perceptual uncertainty. Here we investigate the computational principles that govern reward-based learning and decision making under perceptual uncertainty about environmental states. Based on an integrated perceptual and economic decision-making task where unobservable states governed the reward contingencies, we analyzed behavioral data of 52 human participants. We formalized perceptual uncertainty with a belief state that expresses the probability of task states based on sensory information. Using several Bayesian and Q-learning agent models, we examined to which degree belief states and categorical-choice biases determine human learning and decision making under perceptual uncertainty. We found that both factors influenced participants’ behavior, which was similarly captured in Bayesian-inference and Q-learning models. Therefore, humans dynamically combine uncertain perceptual and reward information during learning and decision making, but categorical choices substantially modulate this integration. The results suggest that categorical commitments to the most likely state of the environment may generally give rise to categorical biases on learning under uncertainty.

Reinforcement learning under uncertainty: expected versus unexpected uncertainty and state versus reward uncertainty

“…We showed that most participants were able to adapt successfully to all three types of uncertainty; they managed to learn the correct state-action mapping under expected state uncertainty (due to noisy visual inputs) and expected reward uncertainty (due to reward being generated from a probability distribution), and managed to adapt to the unexpected uncertainty due to a nonsignalled reversal in the state-action reward contingency. This is in line with previous work that looked at reward-based learning under expected reward uncertainty (Don et al, 2019), expected state uncertainty (Bruckner et al, 2020) and unexpected reward uncertainty (Brown & Steyvers, 2009). Surprisingly, participants collected more rewards in state uncertainty than in reward uncertainty, partly due to their higher exploration rate under the reward uncertainty condition.…”

“…A more recent study showed that people use belief states when learning under state uncertainty, but their learning is modulated by categorical perceptual commitments (Bruckner et al, 2020). As in our experiment, participants had to learn state-response mappings via reward feedback under perceptual uncertainty.…”

“…Another possible explanation for the superiority of the WTA-RL model in our paradigm is that this could depend on the level of state uncertainty, with higher levels requiring belief-state based style of learning to avoid that frequently misinterpreted percepts corrupts, or at least slows down, learning (Bruckner et al, 2020). This is partially supported by our findings in the first experiment (see S.3), where the simple and Bayesian RL models were almost indiscernible at a lower uncertainty threshold (0.65 instead of 0.7).…”

“…While past studies have given some insights into psychological and neural responses to reward variability, little is known about how people learn from reward when the source of uncertainty is the environment's state and rewards themselves are deterministic (though see Bruckner et al, 2020, for a recent attempt to explore reward-based learning under state uncertainty). This situation arises when the agent cannot tell for sure what the current state of the world is: its perceptual system might be noisy (e.g., noise in the brain or in sensors); the observations are ambiguous (e.g.…”

Time pressure changes how people explore and respond to uncertainty

“…Our current results also only examined uncertainty about reward expectations. However, there exist several alternative measures of uncertainty such as confidence 71 , 72 , perceptual uncertainty 73 , 74 , and computational uncertainty induced by cognitive load 75 , all of which could influence exploration behavior in different ways. Thus, we expect future studies to increasingly focus on disentangling different sources of uncertainty and their effects on the exploration-exploitation dilemma.…”