Epidemiologic cohort studies have consistently demonstrated that long-term exposure to ambient fine particles (PM2.5) is associated with mortality. Nevertheless, extrapolating results to understudied locations may involve considerable uncertainty. To explore this issue, this review discusses the evidence for (i) the associated risk of mortality, (ii) the shape of the concentration–response function, (iii) a causal interpretation, and (iv) how the source mix/composition of PM2.5 and population characteristics may alter the effect. The accumulated evidence suggests the following: (i) In the United States, the change in all-cause mortality risk per μg/m3 is about 0.8%. (ii) The concentration–response function appears nonlinear. (iii) Causation is overwhelmingly supported. (iv) Fossil fuel combustion-related sources are likely more toxic than others, and age, race, and income may modify the effect. To illustrate the use of our findings in support of a risk assessment in an understudied setting, we consider Kuwait. However, given the complexity of this relationship and the heterogeneity in reported effects, it is unreasonable to think that, in such circumstances, point estimates can be meaningful. Consequently, quantitative probabilistic estimates, which cannot be derived objectively, become essential. Formally elicited expert judgment can provide such estimates, and this review provides the evidence to support an elicitation.
Dozens of coronavirus (COVID-19) forecasting models have been created; however, little information exists on their performance. Here we examine the performance of nine commonly-used COVID-19 forecasting models, as well as equal- and performance-weighted ensembles, based on their knowledge – i.e., accuracy and precision, and their ‘self-knowledge’ – i.e., ‘calibration’ and ‘information’. Calibration and information are measures commonly employed in structured expert judgment to assess an expert’s ability to meaningfully communicate the extent and limits of their knowledge.1 Data on observed COVID-19 mortality in 4 states, selected to reflect differences in racial composition and COVID-19 case rates, over eight weeks in the summer of 2020 provided the basis for evaluating model predictions.Only two models showed little bias (geometric mean of observed/predicted < 10%) and good precision (geometric standard deviation of observed/predicted < 1.6). Three models demonstrated good calibration and information. However, only one model exhibited superior performance in both dimensions.Nearly all models under-predicted COVID-19 mortality, some quite substantially. Further, model performance depends on racial composition and case rates, and forecasts in the short-term outperform forecasts in the medium-term on all criteria. The performance-weighted ensembles also outperformed the equal-weighted ensemble on all criteria.The ability of models to accurately and precisely predict mortality and the ability of the modelers to provide meaningful characterizations of the uncertainty in their estimates are potentially important to model developers and to those using model output to inform decisions.
Coronavirus disease 2019 (COVID-19) forecasts from over 100 models are readily available. However, little published information exists regarding the performance of their uncertainty estimates (i.e. probabilistic performance). To evaluate their probabilistic performance, we employ the classical model (CM), an established method typically used to validate expert opinion. In this analysis, we assess both the predictive and probabilistic performance of COVID-19 forecasting models during 2021. We also compare the performance of aggregated forecasts (i.e. ensembles) based on equal and CM performance-based weights to an established ensemble from the Centers for Disease Control and Prevention (CDC). Our analysis of forecasts of COVID-19 mortality from 22 individual models and three ensembles across 49 states indicates that—(i) good predictive performance does not imply good probabilistic performance, and vice versa; (ii) models often provide tight but inaccurate uncertainty estimates; (iii) most models perform worse than a naive baseline model; (iv) both the CDC and CM performance-weighted ensembles perform well; but (v) while the CDC ensemble was more informative, the CM ensemble was more statistically accurate across states. This study presents a worthwhile method for appropriately assessing the performance of probabilistic forecasts and can potentially improve both public health decision-making and COVID-19 modelling.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
customersupport@researchsolutions.com
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Copyright © 2025 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.