Reliable, rapid, and remote measurement of metacognitive bias

Abstract

Metacognitive biases have been repeatedly associated with transdiagnostic psychiatric dimensions of ‘anxious-depression’ and ‘compulsivity and intrusive thought’, cross-sectionally. To progress our understanding of the underlying neurocognitive mechanisms, new methods are required to measure metacognition remotely, within individuals over time. We developed a gamified smartphone task designed to measure metacognitive (confidence) bias and investigated its psychometric properties across two studies (N=3410 unpaid citizen scientists, N=52 paid participants). We assessed convergent validity, split-half and test-retest reliability, and identified the minimum number of trials required to capture its clinical correlates. Convergent validity of metacognitive bias was moderate (r(50)=0.64, p<0.001) and it demonstrated excellent split-half reliability (r(50)=0.91, p<0.001). Test-retest reliability was also very high (ICC=0.86, N=110). Anxious-depression was associated with decreased confidence (B=-0.23, SE=0.02, p<0.001), while compulsivity and intrusive thought was associated with greater confidence (B=0.07, SE=0.02, p<0.001). Metacognitive biases in decision-making are highly stable within-session, exhibiting strong reliability and associations with clinical correlates are evident in as few as 40 trials. Meta Mind exemplifies how to adapt established cognitive tasks for within-person, longitudinal assessments in computational psychiatry, facilitating the development and testing of causal models.

Confidence biases in problem gambling

“…1 ), comprising a gamified metacognition task and survey items from which the transdiagnostic symptom dimensions Anxiety-Depression and Compulsivity and Intrusive Thought can be calculated. Additional healthy control participants were drawn from a parallel study that aimed to validate the metacognition task in comparison to a more traditional browser-based version, and which has been published separately and followed the same procedures as the present study ( Fox et al., 2024 ). In this partially overlapping study, half of the participants were randomised to receive the smartphone game first (traditional task second) and of those, participants with a PGSI score of 0 and who met our inclusion criteria (below) were included as healthy controls in the present study.…”

“…Participants were included in the present study if they were based in the UK, Ireland, or the US. To support the demographic matching of the control group and the gambler group, we included healthy control participants from the parallel validation study ( Fox et al., 2024 ), if they were either male or above the age of 20. In the final sample analysed in this study, 18.4% were drawn Fox et al.…”

“…Specifically, we excluded subjects with disparities between the preliminary screening survey and the data they provided to the Neureka app in their self-reported gender or country of residence, or if the age they reported at the two timepoints differed by more than 1 year ( N = 49). Consistent with prior work using this task, we excluded participants if the staircase procedure of the metacognition task failed to produce accuracies in the 0.60–0.85 range ( N = 1) ( Fox et al., 2024 ).…”

“…The Metamind task is a gamified version of the Dot Discrimination Task, a perceptual decision-task frequently used to measure metacognition ( Boldt & Yeung, 2015 ; Rouault et al., 2018 ). The smartphone task was recently validated in a separate study and it has moderate convergent validity with a longer traditional browser-based version ( r (50) = 0.64, p < 0.001) as well as excellent split-half reliability ( r (50) = 0.91, p < 0.001) and very high test-retest reliability (ICC = 0.86, N = 110) ( Fox et al., 2024 ). In Metamind, participants are given the task of controlling a spaceship traversing the brain.…”

“…We focus on mean confidence in this study because the quantification of metacognitive sensitivity and efficiency remains a contested question. This is because current measures provide sub-optimal validity and reliability, and require higher trial numbers than we had available in the metacognition task employed in this study ( Arnold, Johnston, Adie, & Yarrow, 2023 ; Desender, Vermeylen, & Verguts, 2022 ; Fox et al., 2024 ; Guggenmos, 2021 ; Rahnev, 2023 ).…”