Calibration of probabilities: The state of the art to 1980

Lichtenstein, Sarah; Fischhoff, Baruch; Phillips, Lawrence D.

doi:10.1017/cbo9780511809477.023

Cited by 1,343 publications

(670 citation statements)

References 3 publications

Supporting

Mentioning

629

Contrasting

Unclassified

Order By: Relevance

“…The studies, summarized in Table 8, indicate that the geotechnical engineers (Hynes and Van Marke, 1976 ), physicists ( Henrion and Fischhoff, 1986 ) and mechanical engineers ( Cooke et al, 1988) evaluated were overconfident and, as a result, poorly calibrated. The relatively poor surprise index results in these studies are comparable to the poor performance by nonexperts reported by Lichtenstein et al ( 1982 ) (as updated by Morgan and Henrion, 1990 ). In those studies, the surprise indices were on average, between 30% and 40% but ranged from 5% to 50% compared to an ideal surprise index of 2%.…”

Section: Discussionsupporting

confidence: 82%

“…Good calibration in such studies is defined by having the true value for the quantity falling outside the credible interval no more than 2% of the time (i.e., a ''surprise index'' of 2% ). With some exceptions, Lichtenstein et al ( 1982 ) reported surprise indices in the range of 5 -50% indicating that the credible intervals did not contain the truth as often as the subjects predicted.…”

Section: Introductionmentioning

confidence: 87%

“…Most of what we know comes from studies of lay judgments, and suggests that individuals are not well calibrated. Lichtenstein et al (1982 ) and Morgan and Henrion (1990 ) summarized the results from several calibration studies in the literature, where individuals were asked to give confidence intervals for specific quantities (typically 98% confidence intervals ). Good calibration in such studies is defined by having the true value for the quantity falling outside the credible interval no more than 2% of the time (i.e., a ''surprise index'' of 2% ).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene

Walker

Catalano

Hammitt

et al. 2003

J Expo Sci Environ Epidemiol

View full text Add to dashboard Cite

The recent movement of regulatory agencies toward probabilistic analyses of human health and environmental risks has focused greater attention on the quality of the estimates of variability and uncertainty that underlie them. Of particular concern is how uncertainty -a measure of what is not known -is characterized, as uncertainty can play an influential role in analyses of the need for regulatory controls or in estimates of the economic value of additional research. This paper reports the second phase of a study, conducted as an element of the National Human Exposure Assessment Survey ( NHEXAS ), to obtain and calibrate exposure assessment experts' judgments about uncertainty in residential ambient, residential indoor, and personal air benzene concentrations experienced by the nonsmoking, nonoccupationally exposed population in U.S. EPA's Region V. Subjective judgments ( i.e., the median, interquartile range, and 90% confidence interval ) about the means and 90th percentiles of each of the benzene distributions were elicited from the seven experts participating in the study. The calibration or quality of the experts' judgments was assessed by comparing them to the actual measurements from the NHEXAS Region V study using graphical techniques, a quadratic scoring rule, and surprise and interquartile indices. The results from both quantitative scoring methods suggested that, considered collectively, the experts' judgments were relatively well calibrated although on balance, underconfident. The calibration of individual expert judgments appeared variable, highlighting potential pitfalls in reliance on individual experts. In a surprising finding, the experts' judgments about the 90th percentiles of the benzene distributions were better calibrated than their predictions about the means; the experts tended to be overconfident in their ability to predict the means. This paper is also one of the first calibration studies to demonstrate the importance of taking into account intraexpert correlation on the statistical significance of the findings. When the judgments were assumed to be independent, analysis of the surprise and interquartile indices found evidence of poor calibration ( P < 0.05 ). However, when the intraexpert correlation in the study was taken into account, these findings were no longer statistically significant. The analysis further found that the experts' judgments scored better than estimates of Region V benzene concentrations simply drawn from earlier studies of ambient, indoor and personal benzene levels in other U.S. cities. These results suggest the value of careful elicitation of expert judgments in characterizing exposures in probabilistic form. Additional calibration studies need to be undertaken to corroborate and extend these findings.

show abstract

Section: Discussionsupporting

confidence: 82%

Section: Introductionmentioning

confidence: 87%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene

Walker

Catalano

Hammitt

et al. 2003

J Expo Sci Environ Epidemiol

View full text Add to dashboard Cite

show abstract

“…Perfect calibration is indicated by the diagonal line (Lichenstein et al, 1982). Results show that participants in the placebo group are relatively well calibrated.…”

Section: Calibration (Absolute Resolution)mentioning

confidence: 99%

“…Following the procedure commonly used in calibration studies (see Lichenstein et al, 1982), Figure 1 plots the percentage of correct recall against mean JOL ratings, with the probability judgments grouped into 10 preset intervals (0-10, 11-20y91-100; see Koriat, 1997). Perfect calibration is indicated by the diagonal line (Lichenstein et al, 1982).…”

Section: Calibration (Absolute Resolution)mentioning

confidence: 99%

Specific Effects of an Amnesic Drug: Effect of Lorazepam on Study Time Allocation and on Judgment of Learning

Izaute

Bacon

2004

Neuropsychopharmacol

View full text Add to dashboard Cite

We investigated the effects of lorazepam, a benzodiazepine, on the allocation of study time, memory, and judgment of learning, in a cognitive task where the repetition of word presentation was manipulated. The aim was to assess whether lorazepam would affect the learning processes and/or whether the participants would be aware of the amnesic difficulty. A total of 30 healthy volunteers participated in the study, 15 of whom received a capsule containing the lorazepam drug (0.038 mg/kg) and 15 a placebo capsule. First, the accuracy of delayed judgments of learning (JOL) was measured in both groups. For the JOL ratings, results showed that all the participants benefited from word repetition. Although the overall performance was lower in the lorazepam than in the placebo group, the accuracy of the JOL ratings was preserved by the drug. Second, all the participants benefited from the repetition of learning, although the performances of the lorazepam-treated subjects remained lower than those of the placebo participants. The repetition of learning had an effect on JOL in both groups. Finally, the time spent learning each (allocation study time) pair of words was measured. For the placebo group, results revealed that study time decreased significantly with the frequency of presentation. For the lorazepam group, no effect of presentation frequency was found. Overall, our findings suggest that the lorazepam drug has a differential effect on the monitoring and the control processes involved in a learning task. The implications of these findings are discussed within the theoretical framework of metacognition.

show abstract

Medical students' confidence judgments using a factual database and personal memory: A comparison

O'Keefe

Wildemuth

Friedman

1999

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

Calibration of probabilities: The state of the art to 1980

Cited by 1,343 publications

References 3 publications

Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene

Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene

Specific Effects of an Amnesic Drug: Effect of Lorazepam on Study Time Allocation and on Judgment of Learning

Medical students' confidence judgments using a factual database and personal memory: A comparison

Contact Info

Product

Resources

About