Reverse engineering of metacognition

Abstract

The human ability to introspect on thoughts, perceptions or actions − metacognitive ability − has become a focal topic of both cognitive basic and clinical research. At the same time it has become increasingly clear that currently available quantitative tools are limited in their ability to make unconfounded inferences about metacognition. As a step forward, the present work introduces a comprehensive modeling framework of metacognition that allows for inferences about metacognitive noise and metacognitive biases during the readout of decision values or at the confidence reporting stage. The model assumes that confidence results from a continuous but noisy and potentially biased transformation of decision values, described by a confidence link function. A canonical set of metacognitive noise distributions is introduced which differ, amongst others, in their predictions about metacognitive sign flips of decision values. Successful recovery of model parameters is demonstrated, and the model is validated on an empirical data set. In particular, it is shown that metacognitive noise and bias parameters correlate with conventional behavioral measures. Crucially, in contrast to these conventional measures, metacognitive noise parameters inferred from the model are shown to be independent of performance. This work is accompanied by a toolbox (ReMeta) that allows researchers to estimate key parameters of metacognition in confidence datasets.

Simultaneous modeling of choice, confidence, and response time in visual perception.

“…A large number of different measures of metacognitive accuracy exist, some of which are model free, such as Goodman and Kruskal’s Gamma (Nelson, 1984) and the area under Type 2 receiver operating characteristic curve (Fleming et al, 2010). However, there are also measures of metacognitive accuracy that rely on specific confidence models to disentangle between metacognitive accuracy and subjective criteria, for example, meta- d ′/ d ′ (Maniscalco & Lau, 2012), σ meta (Shekhar & Rahnev, 2021), confidence efficiency (Mamassian & de Gardelle, 2022), or metacognitive noise (Guggenmos, 2022). Future studies are needed to investigate how these measures of metacognitive accuracy are related to the parameters of the dynWEV model (or other dynamical models of confidence) to see if measures of metacognitive accuracy are incomplete or even biased due to not accounting for dynamic accumulation processes.…”

GGSDT: A unified signal detection framework for confidence data analysis

“…In other words, the GGSDT is supposed to be a measurement model, being agnostic of specific processes behind varying levels of metacognitive performance. To pinpoint and draw specific conclusions regarding the mechanisms of particular metacognitive behavior, a process model approach (e.g., Fleming & Daw, 2017;Guggenmos, 2022;Shekhar & Rahnev, 2021;Webb et al, 2022) would be required on top of the current measurement framework.…”

“…This provides a strong rationale for treating β as a metacognitive efficiency measure being controlled for objective decision accuracy. The intuition behind this is that metacognitive lapse (putative operation of β) causes greater information loss in the situation of higher objective accuracy (when S1 and S2 distributions are less overlapped), through which the influence of objective accuracy is implicitly taken into account in the β estimate (for more discussion about objective accuracy contamination in metacognitive measurement, see Fleming & Lau, 2014;Guggenmos, 2022).…”

The future of metacognition research: balancing construct breadth with measurement rigor

“…The assumption that Mratio is independent of metacognitive biases (average confidence) has been recently challenged by studies showing that using higher levels of confidence ratings can lead to inflated values of Mratio (Shekhar & Rahnev, 2021a;Xue et al, 2021). Similarly, the assumption that Mratio is performance-independent has been systematically evaluated in both simulation and empirical studies, with nonlinearities in this relationship leading to new model-based metrics with more stable psychometric properties (Barrett, 2013;Guggenmos, 2021Guggenmos, , 2022.Another issue that has come to the fore with several metacognitive measures including Mratio is related to the common usage of staircasing procedures to control first-order performance, with recent work showing that the fluctuations in task difficulty inherent to staircasing can artificially inflate metacognitive efficiency (Rahnev & Fleming, 2019). Finally, hierarchical versions of the meta-d' model have been developed to allow more accurate group-level inference in situations where there is limited data available per subject, such as in clinical studies (Fleming, 2017).…”

“…Another issue is that the meta-d' framework is not a process model of how confidence ratings are generated (Shekhar & Rahnev, 2021a), and thus cannot identify distinct sources of metacognitive inefficiency (Shekhar & Rahnev, 2021b). Thus, just as vision scientists may investigate the different component processes that lead to a particular d', metacognition researchers are increasingly turning to richer computational models to decompose the different stages involved in confidence formation (Bang & Fleming, 2018;Boundy-Singer et al, 2022;Guggenmos, 2022;Shekhar & Rahnev, 2018). Of particular interest here is whether confidence reflects a heuristic such as distance to a decision criterion or bound (Kepecs et al, 2008;Vickers, 1979), or whether it is Bayesian or quasi-Bayesian in being sensitive to uncertainty Aitchison & Lengyel, 2017;Denison et al, 2018;Li & Ma, 2020).…”