Development and Validation of a Machine Learning Model to Estimate Risk of Adverse Outcomes Within 30 Days of Opioid Dispensation

Abstract

ImportanceMachine learning approaches can assist opioid stewardship by identifying high-risk opioid prescribing for potential interventions.ObjectiveTo develop a machine learning model for deployment that can estimate the risk of adverse outcomes within 30 days of an opioid dispensation as a potential component of prescription drug monitoring programs using access to real-world data.Design, Setting, and ParticipantsThis prognostic study used population-level administrative health data to construct a machine learning model. This study took place in Alberta, Canada (from January 1, 2018, to December 31, 2019), and included all patients 18 years and older who received at least 1 opioid dispensation from a community pharmacy within the province.ExposuresEach opioid dispensation served as the unit of analysis.Main Outcomes and MeasuresOpioid-related adverse outcomes were identified from administrative data sets. An XGBoost model was developed on 2018 data to estimate the risk of hospitalization, an emergency department visit, or mortality within 30 days of an opioid dispensation; validation on 2019 data was done to evaluate model performance. Model discrimination, calibration, and other relevant metrics are reported using daily and weekly predictions on both ranked predictions and predicted probability thresholds using all data from 2019.ResultsA total of 853 324 participants represented 6 181 025 opioid dispensations, with 145 016 outcome events reported (2.3%); 46.4% of the participants were men and 53.6% were women, with a mean (SD) age of 49.1 (15.6) years for men and 51.0 (18.0) years for women. Of the outcome events, 77 326 (2.6% pretest probability) occurred within 30 days of a dispensation in the validation set (XGBoost C statistic, 0.82 [95% CI, 0.81-0.82]). The top 0.1 percentile of estimated risk had a positive likelihood ratio (LR) of 28.7, which translated to a posttest probability of 43.1%. In our simulations, the weekly measured predictions had higher positive LRs in both the highest-risk dispensations and percentiles of estimated risk compared with predictions measured daily. Net benefit analysis showed that using machine learning prediction may not add additional benefit over the entire range of probability thresholds.Conclusions and RelevanceThese findings suggest that prescription drug monitoring programs can use machine learning classifiers to identify patients at risk of opioid-related adverse outcomes and intervene on high-risk ranked predictions. Better access to available administrative and clinical data could improve the prediction performance of machine learning classifiers and thus expand opioid stewardship efforts.

Applying AI to Structured Real-World Data for Pharmacovigilance Purposes: Scoping Review (Preprint)

“…When it comes to the studies that succeeded 75% [13][14][15], 3 out of 4 studies used external data to evaluate their model which is contained in the Universality criterion (Supplementary table 4). [21] 2018 No No Yes 0,42 Zhu X et al [22] 2022 No Yes No 0,67 Kidwai-Khan F et al [23] 2022 No Yes No 0,58 Sharma V et al [24] 2022 No No No 0,58…”

Machine learning-based prediction of substance use in adolescents: Derivation and validation in multinational datasets in South Korea, USA, and Norway (Preprint)

“…[35] Following further refinement, this model has potential commercial viability [36], especially when combined with a streamlined self-report questionnaire. The existence of multiple models assessing substance use further attests to the commercial potential of our model [37,38]. Emphasizing characteristics predictive of substance use is essential, suggesting the need for systems to alert parents about potential risks their children might face.…”

Garbage in, Garbage out—Words of Caution on Big Data and Machine Learning in Medical Practice

“…article 1 used a Canadian medication prescribing database involving 853 324 participants to predict 30-day opioid-related adverse events. The reported C statistic of 0.82 indicated good discrimination.…”

“…For example, machine learning and data mining have been used in large administrative databases to predict numerous outcomes, including which patients are at risk for adverse events from opiates. A recent article used a Canadian medication prescribing database involving 853 324 participants to predict 30-day opioid-related adverse events. The reported C statistic of 0.82 indicated good discrimination.…”