Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study

Abstract

BackgroundPythia is an automated, clinically curated surgical data pipeline and repository housing all surgical patient electronic health record (EHR) data from a large, quaternary, multisite health institute for data science initiatives. In an effort to better identify high-risk surgical patients from complex data, a machine learning project trained on Pythia was built to predict postoperative complication risk.Methods and findingsA curated data repository of surgical outcomes was created using automated SQL and R code that extracted and processed patient clinical and surgical data across 37 million clinical encounters from the EHRs. A total of 194 clinical features including patient demographics (e.g., age, sex, race), smoking status, medications, comorbidities, procedure information, and proxies for surgical complexity were constructed and aggregated. A cohort of 66,370 patients that had undergone 99,755 invasive procedural encounters between January 1, 2014, and January 31, 2017, was studied further for the purpose of predicting postoperative complications. The average complication and 30-day postoperative mortality rates of this cohort were 16.0% and 0.51%, respectively. Least absolute shrinkage and selection operator (lasso) penalized logistic regression, random forest models, and extreme gradient boosted decision trees were trained on this surgical cohort with cross-validation on 14 specific postoperative outcome groupings. Resulting models had area under the receiver operator characteristic curve (AUC) values ranging between 0.747 and 0.924, calculated on an out-of-sample test set from the last 5 months of data. Lasso penalized regression was identified as a high-performing model, providing clinically interpretable actionable insights. Highest and lowest performing lasso models predicted postoperative shock and genitourinary outcomes with AUCs of 0.924 (95% CI: 0.901, 0.946) and 0.780 (95% CI: 0.752, 0.810), respectively. A calculator requiring input of 9 data fields was created to produce a risk assessment for the 14 groupings of postoperative outcomes. A high-risk threshold (15% risk of any complication) was determined to identify high-risk surgical patients. The model sensitivity was 76%, with a specificity of 76%. Compared to heuristics that identify high-risk patients developed by clinical experts and the ACS NSQIP calculator, this tool performed superiorly, providing an improved approach for clinicians to estimate postoperative risk for patients. Limitations of this study include the missingness of data that were removed for analysis.ConclusionsExtracting and curating a large, local institution’s EHR data for machine learning purposes resulted in models with strong predictive performance. These models can be used in clinical settings as decision support tools for identification of high-risk patients as well as patient evaluation and care management. Further work is necessary to evaluate the impact of the Pythia risk calculator within the clinical workflow on postoperative outcomes and to optimize th...

Nationwide prediction of type 2 diabetes comorbidities

“…The training set was allocated the first 70% of the individuals (sorted by date of first T2D diagnosis) of the full dataset and used for three-fold cross-validated 56 hyperparameter search. Similarly to work by Corey et al 28 , the remaining (most recently diagnosed) 30% of the full dataset were randomly split into two balanced (the proportion of cases was maintained in each split) subsets: a test set which was allocated 20% of the full dataset to be used for model selection for each outcome and the validation set which was allocated the remaining 10% of the full dataset and exclusively used to present final results. For each model type, the best parameter set was chosen by first repeatedly training the model on two-thirds of the training data and evaluating it on the remaining third, and then averaging accuracy measures across the three runs.…”

“…Machine learning (ML) techniques are increasingly being used to analyze electronic health record data to predict future disease onset or its future course [9][10][11][12][13] . These efforts include prediction of onset and complications of cardiovascular disease [14][15][16][17][18][19][20][21] , onset of T2D [22][23][24][25][26] , onset of kidney disease 27 , as well as prediction of postoperative outcomes [28][29][30][31][32] , birth related outcomes 33,34 , mortality 15,35,36 and hospital readmissions [37][38][39][40][41][42][43] . However, current approaches typically suffer from a number of limitations.…”

Prospective and External Evaluation of a Machine Learning Model to Predict In-Hospital Mortality

“…First and foremost, the technology infrastructure required to run models in real-time had to be built. Fortunately, at our institution, such an infrastructure to automatically curate EHR data was already in place, utilizing native functionality in addition to custom developed technologies 24 . Second, there were many differences in data element names between the retrospective training data and prospective, live EHR data.…”

Real-time utilisation of administrative data in the ED to identify older patients at risk: development and validation of the Dynamic Silver Code

“…Administrative data are increasingly used to this purpose, because of their variety, availability, low cost and accuracy 7–9. Such tools have been applied to screen inpatients,10 11 outpatients12 and free-living subjects5 12 13 for mortality, hospitalisation or disease-specific outcomes, aiming for personalised cures 14–16…”