Calibration of scientific reasoning ability

Abstract

Scientific reasoning ability, the ability to reason critically about the quality of scientific evidence, can help laypeople use scientific evidence when making judgments and decisions. We ask whether individuals with greater scientific reasoning ability are also better calibrated with respect to their ability, comparing calibration for skill with the more widely studied calibration for knowledge. In three studies, participants (Study 1: N = 1022; Study 2: N = 101; and Study 3: N = 332) took the Scientific Reasoning Scale (SRS; Drummond & Fischhoff, 2017), comprised of 11 true–false problems, and provided confidence ratings for each problem. Overall, participants were overconfident, reporting mean confidence levels that were 22.4–25% higher than their percentages of correct answers; calibration improved with score. Study 2 found similar calibration patterns for the SRS and another skill, the Cognitive Reflection Test (CRT), measuring the ability to avoid intuitive but incorrect answers. SRS and CRT scores were both associated with success at avoiding negative decision outcomes, as measured by the Decision Outcomes Inventory; confidence on the SRS, above and beyond scores, predicted worse outcomes. Study 3 added an alternative measure of calibration, asking participants to estimate the number of items answered correctly. Participants were less overconfident by this measure. SRS scores predicted correct usage of scientific information in a drug facts box task and holding beliefs consistent with the scientific consensus on controversial issues; confidence, above and beyond SRS scores, predicted worse drug facts box performance but stronger science‐consistent beliefs. We discuss the implications of our findings for improving science‐relevant decision‐making.

Researchers’ Perceptions of the State of Social and Personality Psychology

“…Researchers in our sample self-reported, on average, being around a quarter-to-a-half a point higher on intellectual humility, on a 5-point scale, than participants across these 16 samples (Mrange = 3.78 -4.10; SDsrange = 0.55 -1.04; see Bowes & Tasimi, 2022;Deffler et al, 2016;Drummond Otten & Fischhoff, 2022;Leary et al, 2017;Meagher, 2022;Stanley et al, 2020;Zedelius et al, 2022). Around ⅙ of the researchers in our sample gave themselves the highest possible ratings on intellectual humility.…”