Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene

Abstract

The recent movement of regulatory agencies toward probabilistic analyses of human health and environmental risks has focused greater attention on the quality of the estimates of variability and uncertainty that underlie them. Of particular concern is how uncertainty -a measure of what is not known -is characterized, as uncertainty can play an influential role in analyses of the need for regulatory controls or in estimates of the economic value of additional research. This paper reports the second phase of a study, conducted as an element of the National Human Exposure Assessment Survey ( NHEXAS ), to obtain and calibrate exposure assessment experts' judgments about uncertainty in residential ambient, residential indoor, and personal air benzene concentrations experienced by the nonsmoking, nonoccupationally exposed population in U.S. EPA's Region V. Subjective judgments ( i.e., the median, interquartile range, and 90% confidence interval ) about the means and 90th percentiles of each of the benzene distributions were elicited from the seven experts participating in the study. The calibration or quality of the experts' judgments was assessed by comparing them to the actual measurements from the NHEXAS Region V study using graphical techniques, a quadratic scoring rule, and surprise and interquartile indices. The results from both quantitative scoring methods suggested that, considered collectively, the experts' judgments were relatively well calibrated although on balance, underconfident. The calibration of individual expert judgments appeared variable, highlighting potential pitfalls in reliance on individual experts. In a surprising finding, the experts' judgments about the 90th percentiles of the benzene distributions were better calibrated than their predictions about the means; the experts tended to be overconfident in their ability to predict the means. This paper is also one of the first calibration studies to demonstrate the importance of taking into account intraexpert correlation on the statistical significance of the findings. When the judgments were assumed to be independent, analysis of the surprise and interquartile indices found evidence of poor calibration ( P < 0.05 ). However, when the intraexpert correlation in the study was taken into account, these findings were no longer statistically significant. The analysis further found that the experts' judgments scored better than estimates of Region V benzene concentrations simply drawn from earlier studies of ambient, indoor and personal benzene levels in other U.S. cities. These results suggest the value of careful elicitation of expert judgments in characterizing exposures in probabilistic form. Additional calibration studies need to be undertaken to corroborate and extend these findings.

Expert Judgment Assessment of the Mortality Impact of Changes in Ambient Fine Particulate Matter in the U.S.

“…As shown in Figure 1, these include explicit criteria for expert selection, a detailed interview protocol, briefing materials provided to experts in advance of the interview, and expert workshops prior to and following the elicitation (7)(8)(9)(10)(11)(12).…”

Incorporating between-expert uncertainty into state-and-transition simulation models for forest restoration

“…Because we relied on expert opinion to parameterize models, it is desirable to validate these models in order to test predictions and to ensure model credibility (e.g. Kilgo et al, 2002;Walker et al, 2003;Shlisky et al, 2005). This might be done by using historical data or aerial photography, or by collecting ongoing monitoring data.…”

Los cirujanos sobreestiman el riesgo de malignidad de los nódulos tiroideos, evaluación de los estimados subjetivos usando un análisis bayesiano

“…To identify the variables to be included in the model, we followed the uncertainty matrix developed by Walker, et al (13). A literature review was conducted to determine the risk factors for malignancy in a thyroid nodule, and the variables that have been reported as clinically useful were selected (6)(7)(8)(14)(15)(16)(17)(18)(19).…”