Naive empiricism and dogmatism in confidence research: A critical examination of the hard–easy effect.

Abstract

Two robust phenomena in research on confidence in one's general knowledge are the overconfidence phenomenon and the hard-easy effect. In this article, the authors propose that the hard-easy effect has been interpreted with insufficient attention to the scale-end effects, the linear dependency, and the regression effects in data and that the continued adherence to the idea of a "cognitive overconfidence bias" is mediated by selective attention to particular data sets. A quantitative review of studies with 2-alternative general knowledge items demonstrates that, contrary to widespread belief, there is (a) very little support for a cognitive-processing bias in these data; (b) a difference between representative and selected item samples that is not reducible to the difference in difficulty; and (c) near elimination of the hard-easy effect when there is control for scale-end effects and linear dependency.

Overconfidence in Interval Estimates.

“…Because they are imperfect, they will naturally tend to be overconfident when they are very confident, but also underconfident when they are very unconfident. Recent research with binary questions has in general supported this view (e.g., Juslin, Winman, & Olsson, 2000;Klayman et al, 1999). Overconfidence may have been the predominant finding in earlier studies because of a tendency for experimenters to construct tests that favored the harder questions in any given domain.…”

The Role of Representative Design in an Ecological Approach to Cognition.

“…Therefore, the impact of representative design can now be studied across a wide range of studies. Recently, P. Juslin, Winman, and Olsson (2000) conducted a review of 130 overconfidence data sets to quantify the effects of representative and unrepresentative item sampling. Figure 4 depicts the over-and underconfidence scores (regressed onto mean confidence) observed in those studies.…”

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

“…Yet these experts lacked incentives rewarding accuracy, training in the use and interpretation of probability scales, and practice, and perhaps most importantly, they lacked timely and unambiguous feedback (Benson and Onkal 1992, Hoelzl and Rustichini 2005, Larrick 2004). Research has found that each of these can help reduce overconfidence, but effects have generally been studied over short time horizons, usually constrained by the duration of laboratory experimental sessions (Soll et al 2016 questions about the degree to which these debiasing effects generalize over longer time horizons and in more consequential domains (Dawes and Mulford 1996, Gigerenzer 1991, Juslin et al 2000.…”