Miss rate neglect in legal evidence

Dahlman, Christian; Zenker, Frank; Sarwar, Farhan

doi:10.1093/lpr/mgw007

Cited by 5 publications

(16 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the one hand, factors such as overweighting of the binary decision (i.e., individuating information) and the underweighting (or neglect) of the base rate were major contributing factors to the errors across all four experiments. These findings are consistent with the large body of earlier studies using this task paradigm as well as other task paradigms that have attributed the errors variously to one or both of these factors (Kahneman and Tversky, 1973;Fischhoff and Bar-Hillel, 1984;Thompson and Schumann, 1987;Koehler, 1996;Baratgin and Noveck, 2000;Fantino, 2004;Barbey and Sloman, 2007;Dahlman et al, 2016;Sanborn and Chater, 2016; also see Koehler (1996) and the accompanying commentaries).…”

Section: Factors That Contribute Significantly To Estimation Errorssupporting

confidence: 90%

Toward a more nuanced understanding of probability estimation biases

Branch

Hegdé

2023

Front. Psychol.

View full text Add to dashboard Cite

In real life, we often have to make judgements under uncertainty. One such judgement task is estimating the probability of a given event based on uncertain evidence for the event, such as estimating the chances of actual fire when the fire alarm goes off. On the one hand, previous studies have shown that human subjects often significantly misestimate the probability in such cases. On the other hand, these studies have offered divergent explanations as to the exact causes of these judgment errors (or, synonymously, biases). For instance, different studies have attributed the errors to the neglect (or underweighting) of the prevalence (or base rate) of the given event, or the overweighting of the evidence for the individual event (‘individuating information’), etc. However, whether or to what extent any such explanation can fully account for the observed errors remains unclear. To help fill this gap, we studied the probability estimation performance of non-professional subjects under four different real-world problem scenarios: (i) Estimating the probability of cancer in a mammogram given the relevant evidence from a computer-aided cancer detection system, (ii) estimating the probability of drunkenness based on breathalyzer evidence, and (iii & iv) estimating the probability of an enemy sniper based on two different sets of evidence from a drone reconnaissance system. In each case, we quantitatively characterized the contributions of the various potential explanatory variables to the subjects’ probability judgements. We found that while the various explanatory variables together accounted for about 30 to 45% of the overall variance of the subjects’ responses depending on the problem scenario, no single factor was sufficient to account for more than 53% of the explainable variance (or about 16 to 24% of the overall variance), let alone all of it. Further analyses of the explained variance revealed the surprising fact that no single factor accounted for significantly more than its ‘fair share’ of the variance. Taken together, our results demonstrate quantitatively that it is statistically untenable to attribute the errors of probabilistic judgement to any single cause, including base rate neglect. A more nuanced and unifying explanation would be that the actual biases reflect a weighted combination of multiple contributing factors, the exact mix of which depends on the particular problem scenario.

show abstract

Section: Factors That Contribute Significantly To Estimation Errorssupporting

confidence: 90%

Toward a more nuanced understanding of probability estimation biases

Branch

Hegdé

2023

Front. Psychol.

View full text Add to dashboard Cite

show abstract

“…Fourth, our study identifies a novel effect that significantly contributes to the estimation errors by both radiologists and non-professional subjects, namely the conditional neglect of false alarm rates. The neglect of the false alarm rate has been previously reported in the context of legal decision-making (Dahlman et al, 2016 ). However, to our knowledge, the present study represents the first report of a conditional neglect of the false alarm rate in any decision-making context.…”

Section: Discussionmentioning

confidence: 88%

“…Our results empirically demonstrate these effects in the context of CAD result interpretation. The neglect of the false alarm rate has been previously reported in the context of legal decision-making (Dahlman et al, 2016 ). However, to our knowledge, the present study represents the first report of a conditional neglect of the false alarm rate in any decision-making context.…”

Section: Discussionmentioning

confidence: 88%

“…Moreover, there is a clear-cut mathematical formulation of the underlying decision problem, making it at least as principled a choice for a task paradigm as any other (see, e.g., (Marewski & Gigerenzer, 2012 )). Second, this paradigm has been used to study similar problems in a diverse array of other realms, including medicine, business, and law, just to name a few (Dahlman et al, 2016 ; Eddy, 2005 ; Kahneman et al, 1982 ; Koehler, 1996 ; Mandel, 2014 ). This makes it easier to compare our empirical results with those from previous studies.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

How well do practicing radiologists interpret the results of CAD technology? A quantitative characterization

et al. 2022

View full text Add to dashboard Cite

Many studies have shown that using a computer-aided detection (CAD) system does not significantly improve diagnostic accuracy in radiology, possibly because radiologists fail to interpret the CAD results properly. We tested this possibility using screening mammography as an illustrative example. We carried out two experiments, one using 28 practicing radiologists, and a second one using 25 non-professional subjects. During each trial, subjects were shown the following four pieces of information necessary for evaluating the actual probability of cancer in a given unseen mammogram: the binary decision of the CAD system as to whether the mammogram was positive for cancer, the true-positive and false-positive rates of the system, and the prevalence of breast cancer in the relevant patient population. Based only on this information, the subjects had to estimate the probability that the unseen mammogram in question was positive for cancer. Additionally, the non-professional subjects also had to decide, based on the same information, whether to recall the patients for additional testing. Both groups of subjects similarly (and significantly) overestimated the cancer probability regardless of the categorical CAD decision, suggesting that this effect is not peculiar to either group. The misestimations were not fully attributable to causes well-known in other contexts, such as base rate neglect or inverse fallacy. Non-professional subjects tended to recall the patients at high rates, even when the actual probably of cancer was at or near zero. Moreover, the recall rates closely reflected the subjects’ estimations of cancer probability. Together, our results show that subjects interpret CAD system output poorly when only the probabilistic information about the underlying decision parameters is available to them. Our results also highlight the need for making the output of CAD systems more readily interpretable, and for providing training and assistance to radiologists in evaluating the output.

show abstract

“…They think less about the miss rate, and are less attentive to circumstances that affect the miss rate. A legal decision maker who disregards the miss rate, underestimates the importance of the miss rate, or overlooks circumstances that affect the miss rate commits miss rate neglect (Dahlman, Zenker, & Sarwar, 2016).…”

Section: Fallacies and Biasesmentioning

confidence: 99%