Real-world Antimicrobial Stewardship Experience in a Large Academic Medical Center: Using Statistical and Machine Learning Approaches to Identify Intervention “Hotspots” in an Antibiotic Audit and Feedback Program

Abstract

Background Prospective audit with feedback (PAF) is an impactful strategy for antimicrobial stewardship program (ASP) activities. However, because PAF requires reviewing large numbers of antimicrobial orders on a case-by-case basis, PAF programs are highly resource-intensive. The current study aimed to identify predictors of ASP intervention (i.e., feedback), and to build models to identify orders that can be safely bypassed from review, to make PAF programs more efficient. Methods We performed a retrospective cross-sectional study of inpatient antimicrobial orders reviewed by the University of Maryland Medical Center’s PAF program between 2017–2019. We evaluated the relationship between antimicrobial and patient characteristics with ASP intervention using multivariable logistic regression models. Separately, we built prediction models for ASP intervention using statistical and machine learning approaches and evaluated performance on held-out data. Results Across 17,503 PAF reviews, 4,219 (24%) resulted in intervention. In adjusted analyses, a clinical pharmacist on the ordering unit or receipt of an ID consult were associated with 17% and 56% lower intervention odds, respectively (aORs 0.83 and 0.44, P values ≤ 0.001). Fluoroquinolones had the highest adjusted intervention odds (aOR 3.22, 95% CI: 2.63–3.96). A machine learning classifier (C-statistic 0.76) reduced reviews by 49% while achieving 78% sensitivity. A “workflow simplified” regression model that restricted to antimicrobial class and clinical indication variables, two strong machine-learning-identified predictors, reduced reviews by one-third while achieving 81% sensitivity. Conclusions Prediction models substantially reduced PAF review caseloads while maintaining high sensitivities. Our results and approach may offer a blueprint for other ASPs.

Improving the Efficiency of Antimicrobial Stewardship Action in Acute Care Facilities

“…The last approach to prioritization is the use of predictive modeling and machine learning, which provide information on what characteristics predict AS intervention in a desired population [ 26–29 ]. For example, a newly staffed stewardship program with a dedicated stewardship pharmacist at a small community hospital may not be immediately equipped with the knowledge of current antibiotic use challenges.…”

“…A reasonable first approach to inform stewardship practice is to perform PAF on preselected and frequently prescribed antibiotics at 48 to 72 hours and document guideline compliance, details of the intervention, and whether it was accepted. Using the PAF data with linked variables that can be easily identified and sorted at the time of an alert (eg, antibiotic type, indication, hospital service), the AS member or technology specialist can enter them into prebuilt models available in the literature [ 26 ]. Discrete variables of importance can be identified for future targets with the goal to decrease the case load while not appreciably lowering the sensitivity of actionable alerts.…”

Brave New World of Artificial Intelligence: Its Use in Antimicrobial Stewardship—A Systematic Review

“…A total of 4658 citations were identified from the three databases and, after removing the duplicates, 2839 were eligible for screening. A total of 1086 articles were assessed for eligibility and eighteen [20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37] were included in this systematic review (Figure 1). Most studies were excluded because they did not study the application of machine learning models nor their predictive performance or because they were not applied to hospital inpatients and outpatients with infections, such as studies in vitro or regarding drug development.…”

“…All the studies were rated as being of "fair quality" by the NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies; fourteen studies were rated as 57.1% and four [22,26,32,35] were rated as 64.3%. The participation rate, variation in amount or level of exposure, and loss to follow-up criteria were not applied to any of the studies.…”

“…Only one study [35] provided a sample size justification or power description. No study reported information about blinding the assessors, and only three studies [22,26,32] met the criterion on the statistical adjustments of potential confounding variables. The answer to each of the fourteen criteria, as well as the quality rating, are available in Table 3.…”