Optimal Triage for COVID-19 Patients Under Limited Health Care Resources With a Parsimonious Machine Learning Prediction Model and Threshold Optimization Using Discrete-Event Simulation: Development Study

Kim, Jeong Min; Lim, Hyun; Ahn, Jae‐Hyeon; Lee, Kyoung Hwa; Lee, Kwang Suk; Koo, Kyo Chul

doi:10.2196/32726

Cited by 13 publications

(5 citation statements)

References 28 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We exhaustively developed and simultaneously validated all possible combinations of candidate feature subsets and modeling algorithms (Figure S7 in Multimedia Appendix 1). Contrary to our study design, previous prognostic studies relied on a single feature selection approach (clinical consensus, least absolute shrinkage and selection operator regression, recursive feature elimination, and sequential forward selection) [26,28,32,34,35]. Since there is no one-size-fits-all solution in the model fine-tuning process [36], this comprehensive modeling procedure can provide engineering value in attaining optimal prediction with parsimonious feature usage.…”

Section: Principal Findingsmentioning

confidence: 99%

Development and Validation of a Robust and Interpretable Early Triaging Support System for Patients Hospitalized With COVID-19: Predictive Algorithm Modeling and Interpretation Study

Baek,

Jeong,

Kim

et al. 2024

J Med Internet Res

View full text Add to dashboard Cite

Background Robust and accurate prediction of severity for patients with COVID-19 is crucial for patient triaging decisions. Many proposed models were prone to either high bias risk or low-to-moderate discrimination. Some also suffered from a lack of clinical interpretability and were developed based on early pandemic period data. Hence, there has been a compelling need for advancements in prediction models for better clinical applicability. Objective The primary objective of this study was to develop and validate a machine learning–based Robust and Interpretable Early Triaging Support (RIETS) system that predicts severity progression (involving any of the following events: intensive care unit admission, in-hospital death, mechanical ventilation required, or extracorporeal membrane oxygenation required) within 15 days upon hospitalization based on routinely available clinical and laboratory biomarkers. Methods We included data from 5945 hospitalized patients with COVID-19 from 19 hospitals in South Korea collected between January 2020 and August 2022. For model development and external validation, the whole data set was partitioned into 2 independent cohorts by stratified random cluster sampling according to hospital type (general and tertiary care) and geographical location (metropolitan and nonmetropolitan). Machine learning models were trained and internally validated through a cross-validation technique on the development cohort. They were externally validated using a bootstrapped sampling technique on the external validation cohort. The best-performing model was selected primarily based on the area under the receiver operating characteristic curve (AUROC), and its robustness was evaluated using bias risk assessment. For model interpretability, we used Shapley and patient clustering methods. Results Our final model, RIETS, was developed based on a deep neural network of 11 clinical and laboratory biomarkers that are readily available within the first day of hospitalization. The features predictive of severity included lactate dehydrogenase, age, absolute lymphocyte count, dyspnea, respiratory rate, diabetes mellitus, c-reactive protein, absolute neutrophil count, platelet count, white blood cell count, and saturation of peripheral oxygen. RIETS demonstrated excellent discrimination (AUROC=0.937; 95% CI 0.935-0.938) with high calibration (integrated calibration index=0.041), satisfied all the criteria of low bias risk in a risk assessment tool, and provided detailed interpretations of model parameters and patient clusters. In addition, RIETS showed potential for transportability across variant periods with its sustainable prediction on Omicron cases (AUROC=0.903, 95% CI 0.897-0.910). Conclusions RIETS was developed and validated to assist early triaging by promptly predicting the severity of hospitalized patients with COVID-19. Its high performance with low bias risk ensures considerably reliable prediction. The use of a nationwide multicenter cohort in the model development and validation implicates generalizability. The use of routinely collected features may enable wide adaptability. Interpretations of model parameters and patients can promote clinical applicability. Together, we anticipate that RIETS will facilitate the patient triaging workflow and efficient resource allocation when incorporated into a routine clinical practice.

show abstract

Section: Principal Findingsmentioning

confidence: 99%

Development and Validation of a Robust and Interpretable Early Triaging Support System for Patients Hospitalized With COVID-19: Predictive Algorithm Modeling and Interpretation Study

Baek,

Jeong,

Kim

et al. 2024

J Med Internet Res

View full text Add to dashboard Cite

show abstract

“…al. (11) used a discrete event simulation approach, with accompanying demand models for COVID-19 patients, to train a bespoke machine learning model for prioritizing critical care beds, but they did not assess the performance of proposed triage protocols. None of the previously reported simulation methods attempted to evaluate the tradeoff between saving the most lives to discharge and saving the most life-years.…”

Section: S3mentioning

confidence: 99%

Investigating Ethical Tradeoffs in Crisis Standards of Care through Simulation of Ventilator Allocation Protocols

Herington,

Shand,

Holden-Wiltse

et al. 2024

Preprint

View full text Add to dashboard Cite

Introduction: Arguments over the appropriate Crisis Standards of Care (CSC) for public health emergencies often assume that there is a tradeoff between saving the most lives, saving the most life-years, and preventing racial disparities. However, these assumptions have rarely been explored empirically. To quantitatively characterize possible ethical tradeoffs, we aimed to simulate the implementation of five proposed CSC protocols for rationing ventilators in the context of the COVID-19 pandemic. Methods: A Monte Carlo simulation was used to estimate the number of lives saved and life-years saved by implementing clinical acuity-, comorbidity- and age-based CSC protocols under different shortage conditions. This model was populated with patient data from 3707 adult admissions requiring ventilator support in a New York hospital system between April 2020 and May 2021. To estimate lives and life-years saved by each protocol, we determined survival to discharge and estimated remaining life expectancy for each admission. Results: The simulation demonstrated stronger performance for age- and comorbidity-sensitive protocols. For a capacity of 1 bed per 2 patients, ranking by age bands saves approximately 28.7 lives and 3408 life-years per thousand patients, while ranking by Sequential Organ Failure Assessment (SOFA) bands saved the fewest lives (13.2) and life-years (416). For all protocols, we observed a positive correlation between lives saved and life-years saved. For all protocols except lottery and the banded SOFA, significant disparities in lives saved and life-years saved were noted between White non-Hispanic, Black non-Hispanic, and Hispanic sub-populations. Conclusion: While there is significant variance in the number of lives saved and life-years saved, we did not find a tradeoff between saving the most lives and saving the most life-years. Moreover, concerns about racial discrimination in triage protocols require thinking carefully about the tradeoff between enforcing equality of survival rates and maximizing the lives saved in each sub-population.

show abstract

“…Moreover, we conducted comparative analyses with 14 well-known single classifiers, including Logistic Regression and XGBoost models. These models have previously been applied in studies utilizing the KDBC dataset to predict COVID-19 severity in South Korea [23,49,50]. To compare our results with those of more advanced methods, we also conducted experiments with two AutoML approaches: PyCaret, and H2O.ai [51,52].…”

Section: Performance Of the Prediction Modelsmentioning

confidence: 99%

Adaptive Stacking Ensemble Techniques for Early Severity Classification of COVID-19 Patients

Kim,

Ju,

Seok

et al. 2024

Applied Sciences

View full text Add to dashboard Cite

During outbreaks of infectious diseases, such as COVID-19, it is critical to rapidly determine treatment priorities and identify patients requiring hospitalization based on clinical severity. Although various machine learning models have been developed to predict COVID-19 severity, most have limitations, such as small dataset sizes, the limited availability of clinical variables, or a constrained classification of severity levels by a single classifier. In this paper, we propose an adaptive stacking ensemble technique that identifies various COVID-19 patient severity levels and separates them into three formats: Type 1 (low or high severity), Type 2 (mild, severe, critical), and Type 3 (asymptomatic, mild, moderate, severe, fatal). To enhance the model’s generalizability, we utilized a nationwide dataset from the South Korean government, comprising data from 5644 patients across over 100 hospitals. To address the limited availability of clinical variables, our technique employs data-driven strategies and a proposed feature selection method. This ensures the availability of clinical variables across diverse hospital environments. To construct optimal stacking ensemble models, our technique adaptively selects candidate base classifiers by analyzing the correlation between their predicted outcomes and performance. It then automatically determines the optimal multi-layer combination of base and meta-classifiers using a greedy search algorithm. To further improve the performance, we applied various techniques, including imputation of missing values and oversampling. The experimental results demonstrate that our stacking ensemble models significantly outperform existing single classifiers and AutoML approaches, with improvements of 6.42% and 8.86% in F1 and AUC scores for Type 1, 9.59% and 6.68% for Type 2, and 11.94% and 9.24% for Type 3, respectively. Consequently, our approach improves the prediction of COVID-19 severity levels and potentially assists frontline healthcare providers in making informed decisions.

show abstract

Optimal Triage for COVID-19 Patients Under Limited Health Care Resources With a Parsimonious Machine Learning Prediction Model and Threshold Optimization Using Discrete-Event Simulation: Development Study

Cited by 13 publications

References 28 publications

Development and Validation of a Robust and Interpretable Early Triaging Support System for Patients Hospitalized With COVID-19: Predictive Algorithm Modeling and Interpretation Study

Development and Validation of a Robust and Interpretable Early Triaging Support System for Patients Hospitalized With COVID-19: Predictive Algorithm Modeling and Interpretation Study

Investigating Ethical Tradeoffs in Crisis Standards of Care through Simulation of Ventilator Allocation Protocols

Adaptive Stacking Ensemble Techniques for Early Severity Classification of COVID-19 Patients

Contact Info

Product

Resources

About