The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards

Abstract

Problem framing is critical to developing risk prediction models because all subsequent development work and evaluation takes place within the context of how a problem has been framed and explicit documentation of framing choices makes it easier to compare evaluation metrics between published studies. In this work, we introduce the basic concepts of framing, including prediction windows, observation windows, window shifts and event-triggers for a prediction that strongly affects the risk of clinician fatigue caused by false positives. Building on this, we apply four different framing structures to the same generic dataset, using a sepsis risk prediction model as an example, and evaluate how framing affects model performance and learning. Our results show that an apparently good model with strong evaluation results in both discrimination and calibration is not necessarily clinically usable. Therefore, it is important to assess the results of objective evaluations within the context of more subjective evaluations of how a model is framed.

Machine learning for real-time aggregated prediction of hospital admission for emergency patients

“…Our pipeline was designed to generate bed-level predictions from real-time patient-level data streams. We have four prediction times in the day and use data from an observation window to make predictions about the number of admissions in prediction windows of 4 and 8 hours after each prediction time ( italics refer to the terminology of Lauritsen et al 22 ). We constructed the aggregate predictions in a series of seven steps (see Fig.…”

“…Building a model for implementation involves several additional challenges to those encountered when simply optimising the technical performance of a prediction model. These include preparing training examples of incomplete visits from historic data in which visits have been completed 21 , making decisions about the temporal framing of the model (for example, at what point in the visit to check if the outcome of interest has occurred) 22 , and planning for a drift in model performance over time 23 . Models for real-time prediction have been trained in clinical contexts such as circulatory failure in critical care 24 and post-operative complications 25 , 26 .…”

Machine Learning for Real-Time Aggregated Prediction of Hospital Admission for Emergency Patients

“…Our pipeline was designed to generate bed-level predictions from real-time patient-level data streams. We have four prediction times in the day and use data from an observation window to make predictions about the number of admissions in prediction windows of 4 and 8 hours after each prediction time ( italics refer to the terminology of Lauritsen et al 23 ). We constructed the aggregate predictions in a series of seven steps (see Figure 1).…”

“…Using Lauritsen et al’s 23 terminology, samples were left-aligned for training and right-aligned for prediction, as shown in Figure 7. A series of 12 models, each trained on successively longer elapsed times in ED were created.…”

What is new in hemodynamic monitoring and management?

“…Still, a subsequent clinical implementation of the risk spike algorithm is likely to perform dramatically different from the results obtained with a case-control design, because the case-control design applied to time-series is inherently contributing a temporal bias in the analysis [ 30 ]. Applying such a design is therefore unlikely to bridge the gap between a retrospectively developed algorithm and real-world clinical implementation [ 31 ].…”