Measuring metacognitive performance: type 1 performance dependence and test-retest reliability

Abstract

Research on metacognition—thinking about thinking—has grown rapidly and fostered our understanding of human cognition in healthy individuals and clinical populations. Of central importance is the concept of metacognitive performance, which characterizes the capacity of an individual to estimate and report the accuracy of primary (type 1) cognitive processes or actions ensuing from these processes. Arguably one of the biggest challenges for measures of metacognitive performance is their dependency on objective type 1 performance, although more recent methods aim to address this issue. The present work scrutinizes the most popular metacognitive performance measures in terms of two critical characteristics: independence of type 1 performance and test-retest reliability. Analyses of data from the Confidence Database (total N = 6912) indicate that no current metacognitive performance measure is independent of type 1 performance. The shape of this dependency is largely reproduced by extending current models of metacognition with a source of metacognitive noise. Moreover, the reliability of metacognitive performance measures is highly sensitive to the combination of type 1 performance and trial number. Importantly, trial numbers frequently employed in metacognition research are too low to achieve an acceptable level of test-retest reliability. Among common task characteristics, simultaneous choice and confidence reports most strongly improved reliability. Finally, general recommendations about design choices and analytical remedies for studies investigating metacognitive performance are provided.

Are you confident enough to act? Individual differences in action control are associated with post-decisional metacognitive bias

“…In this light, state-oriented individuals might only express lower confidence while not truly experiencing it. Such report bias [ 121 ] could be driven by factors including modesty or excessive self-monitoring [ 122 ]. Finally, a more mechanistic account of metacognitive bias on cognitive and neural levels is needed to understand how the confidence gap arises.…”

Confidence as a noisy decision reliability estimate

“…We found that this model provides an excellent account of choice-confidence data reported in a large set of previously published studies 23–28 . Our analysis suggests that meta-uncertainty provides a better metric for metacognitive ability than the non-process-model based alternatives that currently prevail in the literature 13,15 . Specifically, meta-uncertainty has higher test-retest reliability, is less affected by discrimination ability and response bias, and has comparable cross-domain generalizability.…”

“…It has long been known that humans and other animals can meaningfully introspect about the quality of their decisions and actions [5][6][7]31,48 . Quantifying this ability has remained a significant challenge, even for simple binary decision-making tasks 12,13,15,28,40,41 . The core problem is that observable choice-confidence data reflect metacognitive ability as well as task difficulty and response bias.…”

“…Quantifying a subject’s ability to introspect about the correctness of a decision is a challenging problem 12–14 . There exists no generally agreed-upon method 15 . Even in the simplest decision-making tasks, several distinct factors influence a subject’s confidence reports.…”

Episodic and semantic feeling-of-knowing in aging: a systematic review and meta-analysis

“…In the literature on confidence judgments, two main methods are used to achieve such control. The first is to use a metacognitive sensitivity measure that takes into account first-order performance such as the meta-d'/d' ratio (Fleming, 2017;Maniscalco & Lau, 2012) also known as metacognitive efficiency even though small dependencies between d' and the meta-d'/d' ratio also exists particularly for low first-order performance (Guggenmos, 2021) which can be problematic with respect to the memory impairment in aging. However, as gamma is mainly influenced by bias and guessing and as guessing plays a very much reduced role in recall, we suggest that gamma remains a reliable measure in this context, such as for JOLs and Ease-Of-Learning.…”