How experts' own inconsistency relates to their confidence and between-expert disagreement

Abstract

Data and code to reproduce the analyses reported in this paper are publicly available at https://osf.io/ chgu5. For more information see the Data availability statement at the end of the paper. 1 the model's predictions in two real-world datasets: diagnosticians rating the same mammograms [19] or images of the lower spine [20] twice. To preview one major insight not anticipated by previous accounts of expert inconsistency: Cases on which there was clear between-expert agreement were associated with highly confident initial decisions that were unlikely to change-independent of whether the experts' consensus decision was correct or incorrect.A model linking experts' inconsistency and confidence to a case's ambiguity A fundamental process assumed by many models of cognition, judgment, and decision making is that individuals sample evidence from their environment or memory when making a decision [12-14, 18, 21]. This sampled evidence determines both the decision and the confidence in that decision [18,21,22]. The assumption is that an individual samples several pieces of evidence ("cues") and selects the option for which there is stronger evidence. The more clearly the evidence points to the option, the more confident the individual will be in the accuracy of their decision. In particular, making a second decision about the same case is seen as equivalent to drawing a second sample of evidence. Because the sampling process is probabilistic, the evidence in the second sample can differ from that in the first sample-as can the corresponding decision (e.g., "cancer" vs. "no cancer") and the individual's confidence in that decision.

Harnessing the Wisdom of the Confident Crowd in Medical Image Decision-making

“…Importantly, studies have shown that WoC generalizes to medical image decision making (Juni & Eckstein, 2017;Kurvers et al, 2016;Wolf et al, 2015). Additionally, there has been recent interest in the relationship between accuracy and confidence in expert medical image decision making (Litvinova et al, 2019). In this paper, we build on both lines of inquiry and examine whether incorporating confidence judgments into decision aggregation algorithms improves accuracy in medical image diagnosis for both novices and experts.…”

Group decisions based on confidence weighted majority voting

“…One limitation that our well-controlled setting cannot account for is situations in which individuals consensually reach incorrect decisions with high confidence (see Koriat 2015Koriat , 2017Litvinova et al 2019). In such situations, confidences toward the incorrect decision are aggregated and can lead to high group confidences toward incorrect decisions.…”

“…In particular, MV overlooks that individuals can estimate how accurate their own decisions are in many situations (Brenner et al 1996;Fleming et al 2012;Griffin and Tversky 1992;Martins 2006;Zehetleitner and Rausch 2013;Regenwetter et al 2014) even though there are also situations in which they cannot (Klein and Epley 2015;Koriat 2012bKoriat , 2017Litvinova et al 2019). When reliable confidence estimates are available, they can influence real group discussions: It is plausible that individuals share their sense of confidence during group interactions (Bang et al 2014) such that votes from confident individuals are weighted more than those of less confident individuals.…”

Pooling decisions decreases variation in response bias and accuracy

“…The importance of outcome variation in forecasts, judgments and decisions has been pointed out across the behavioral sciences (Kahneman et al, 2016;Litvinova et al, 2019;Stewart, 2001). For example, in the literature on optimal portfolio selection, the importance of taking into account both the mean rate and the variance (i.e., risk) of returns on securities is well established (Brealey et al, 2012); likewise, in the literature on forecasting (Hibon and Evgeniou, 2005;Lichtendahl and Winkler, 2020) and machine learning (Kuncheva, 2014), aggregation has been shown to reduce risk.…”