Three Critical Questions That Should Be Asked Before Using Prediction Models for Clinical Decision Support

Abstract

To paraphrase Yogi Berra and perhaps others, 1 prediction is hard, especially about the future. Chaudhary and colleagues 2 therefore should be commended for producing a moderately accurate prediction model of sustained postoperative opioid use. Consistent with current standards, 3 they transparently reported the model development process and coefficients. They also translated the model into a practical and accessible scoring system, the Stopping Opioids After Surgery (SOS) score, making it more likely to be used for discharge planning. Others who develop prediction models should emulate these features. Clinical prediction models like the SOS score 2 are intended to inform treatment decisions for individual patients. Therefore, the same rules of evidence and skepticism should apply as for all health care interventions. Because of the complex and technical nature of prediction model development and evaluation, being a critical user of these models is often more difficult than producing the models in the first place. Overall accuracy statistics are important, but they are only a small fraction of what the critical user should consider. In this short commentary, I offer only 3 of the many important questions critical users should ask before using or implementing the SOS score or any other clinical prediction model.

Sensitivity and Specificity of a Short Scale for Assessing Psychological Violence in Peruvian Health Professionals

“…b) The results on ROC curves reported in other studies [3,19] provide conclusive support for the PVS-Health scale because of the discriminative capability (AUC = 0.89 versus limit AUC > 0.80) [19] and because of the randomness of selection, which are adequate in both studies [3,19]. c) Compared to a Pakistani stress questionnaire (AUC = 0.64; kappa = 0.84 [6] versus k = -0.766; AUC =0.89), the use of the PVS-Health scale in clinical and legal settings is stronger given the convergence of stress as an underlying factor [6].…”

“…The instruments used in Peru to assess PVW at different levels of health care are limited because they do not assess utility or classi cation, which hinders the achievement of optimal standards for psychological tests [17,18]. These limitations lead to the necessity of an instrument that exhibits utility, good discriminatory capability, and randomness to classify workers exposed to PVW [19]. In addition, it is necessary to standardize criteria for the interpretation of results to identify, evaluate, and compare the prevalence of PVW; determine corrective actions; and establish baselines for mental health at work.…”

Comparison of Multivariable Logistic Regression and Machine Learning Models for Predicting Bronchopulmonary Dysplasia or Death in Very Preterm Infants

“…A 2019 commentary observed that “the critical user of prediction models should look for evidence that the reported overall accuracy of models still applies for subgroups of patients with very different input or outcome prevalence.” ( 56 ). The incidence of BPD is much higher in younger GA groups, particularly among those born prior to 29 weeks of gestation ( 57 ).…”

Engineering Approaches for Addressing Opioid Use Disorder in the Community

“…Note that the potential of bias and other ethical challenges arising from the use of machine learning approaches in health care is well documented (29). In response, Harris (30) posed three important questions that providers should ask before using clinical prediction models: (a) Does the outcome have the same meaning for all patients? (b) Is the model accurate for subgroups?…”