Neural and computational underpinnings of biased confidence in human reinforcement learning

Abstract

While navigating a fundamentally uncertain world, humans and animals constantly produce subjective confidence judgments, thereby evaluating the probability of their decisions, actions or statements being correct. Confidence typically correlates with neural activity positively in a ventromedial-prefrontal (VMPFC) network and negatively in a dorsolateral and dorsomedial prefrontal network. Here, combining fMRI with a reinforcement-learning paradigm, we leverage the fact that humans are more confident in their choices when seeking gains than avoiding losses to reveal a functional dissociation: whereas the dorsal prefrontal network correlates negatively with a condition-specific confidence signal, the VMPFC network positively encodes task-wide confidence signal incorporating the valence-induced bias. Challenging dominant neuro-computational models, we found that decision-related VMPFC activity better correlates with confidence than with option-values inferred from reinforcement-learning models. Altogether, these results identify the VMPFC as a key node in the neuro-computational architecture that builds global feeling-of-confidence signals from latent decision variables and contextual biases during reinforcement-learning.

High overall values mitigate gaze-related effects in perceptual and preferential choices

“…Another alternative explanation for our findings could be that participants are motivated by the attractive options, which may affect the efficiency of information processing and confidence in choosing the better option. Indeed, we found that self-report confidence was positively related to OV (Supplementary FigureS1), which aligns with the results of several recent studies (Lebreton et al, 2018;Salem-Garcia et al, 2023;Ting et al, 2023). Furthermore, we found that not only RT but also the non-decision time (NDT), which is independent of evidence accumulation, was lowest in the high OV condition (Table 2).…”

The neural representation of metacognition in preferential decision‐making

“…A growing body of evidence suggests that confidence is linearly encoded in vmPFC (De Martino et al, 2013; Rouault et al, 2022; Ting et al, 2023). Consistent with this, results showed that activity in vmPFC was modulated by confidence (SVC with p < .05 FEW‐corrected at peak‐level) (Figure 4a).…”

“…Unlike perceptual decisions, there are few investigations of metacognition in value‐based decisions, and most have focused on searching for the correlates of confidence levels rather than qualify of metacognition (Fleming & De Martino, 2014; Lopez‐Persem et al, 2020; Ting et al, 2023). It has been established that confidence is automatically aggregated with the stimulus value signal in the ventromedial prefrontal cortex (vmPFC) (De Martino et al, 2013; Lebreton et al, 2015; Lopez‐Persem et al, 2016).…”

Learning and metacognition under volatility in GD: Lower learning rates and distorted coupling between action and confidence

“…Recent investigations in healthy populations have begun to elucidate the relationship between learning biases and confidence biases ( Lebreton, Bacily, Palminteri, & Engelmann, 2019 ; Salem-Garcia, Palminteri, & Lebreton, 2023 ; Ting, Salem-Garcia, Palminteri, & Engelmann, 2023 ). These studies have shown that individuals tend to be more confident when learning to seek gains as opposed to avoiding losses.…”

“…These studies have shown that individuals tend to be more confident when learning to seek gains as opposed to avoiding losses. This 'valence-induced confidence bias' has been linked to reduced context-dependent learning, while a general overconfidence bias correlated with a confirmatory learning bias ( Salem-Garcia et al., 2023 ; Ting et al., 2023 ). Applying this framework to GD, one could hypothesize that in an incentivized reinforcement learning task, GD patients would exhibit both elevated confidence and a more pronounced valence-induced confidence bias.…”