Reinforcement biases subsequent perceptual decisions when confidence is low, a widespread behavioral phenomenon

Abstract

Learning from successes and failures often improves the quality of subsequent decisions. Past outcomes, however, should not influence purely perceptual decisions after task acquisition is complete since these are designed so that only sensory evidence determines the correct choice. Yet, numerous studies report that outcomes can bias perceptual decisions, causing spurious changes in choice behavior without improving accuracy. Here we show that the effects of reward on perceptual decisions are principled: past rewards bias future choices specifically when previous choice was difficult and hence decision confidence was low. We identified this phenomenon in six datasets from four laboratories, across mice, rats, and humans, and sensory modalities from olfaction and audition to vision. We show that this choice-updating strategy can be explained by reinforcement learning models incorporating statistical decision confidence into their teaching signals. Thus, reinforcement learning mechanisms are continually engaged to produce systematic adjustments of choices even in well-learned perceptual decisions in order to optimize behavior in an uncertain world.

Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task

“…Previous studies have shown that animals shift their decision boundary even without reward amount manipulations in perceptual decision tasks ( Lak et al, 2020a ). These shifts occur on a trial-by-trial basis, following a win-stay strategy, choosing the same side when that side was associated with reward in the previous trial, particularly when the stimulus was more ambiguous ( Lak et al, 2020a ). In the current task design, however, the optimal bias is primarily determined by the sizes of reward (more specifically, which side delivered a big or small reward) which stays constant across trials within a block.…”

Opposite effects of choice history and evidence history resolve a paradox of sequential choice bias

“…Previous studies of the influence of confidence on choice repetition in forced-choice paradigms have often used proxy measures for confidence, such as pupil dilation, response times, and evidence strength. Notably, a recent study by Lak et al (2020) reported increased choice repetition biases following decisions based on weak sensory evidence and used this sensory evidence strength as a proxy for decision confidence. In disagreement with the explanation by Lak et al (2020) , that low decision confidence boosts choice repetition, we found that low decision confidence, as measured with subjective reports and reaction times, was clearly associated with a decreased probability to repeat the previous choice.…”

Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task

“…Previous studies have shown that animals shift their decision boundary even without reward amount manipulations in perceptual decision tasks (Lak et al, 2020a). These shifts occur on a trial-by-trial basis, following a win-stay strategy, choosing the same side when that side was associated with reward in the previous trial, particularly when the stimulus was more ambiguous (Lak et al, 2020a). In the current task design, however, the optimal bias is primarily determined by the sizes of reward (more specifically, which side delivered a big or small reward) which stays constant across trials within a session.…”