Assessing the effects of data drift on the performance of machine learning models used in clinical sepsis prediction

Detecting changes in the performance of a clinical machine learning tool over time

“…Rahmani and colleagues simulated multiple scenarios in which performance drift of their sepsis prediction tool may occur and showed that retraining methods could significantly improve performance afterwards. 12 Parikh et al. investigated the impact of data shifts related to the COVID-19 pandemic on a mortality prediction algorithm using time-series analyses.…”

Predictive Modelling of Linear Growth Faltering Among Pediatric Patients with Diarrhea in Rural Western Kenya: An Explainable Machine Learning Approach

“…Data-driven predictive models could be leveraged to this end and a number of research efforts exist in the prediction of LGF among children with diarrhea [9,10]. While the existing models provide a valuable starting point, shifts in the study population over time may affect the predictive performance of these models [11,12]. Moreover, development of new models using more recent and pertinent data offers the opportunity to improve model performance and capture new perspectives and insights into this public health problem.…”

Children are small adults (when properly normalized): Transferrable/generalizable sepsis prediction

“…Sepsis is a pathological manifestation of the body's acute inflammatory response to infection and injury, and, despite decades of research, continues to have a significant mortality [ [1] , [2] , [3] ]. Specifically, pediatric sepsis has a significant health impact world-wide [ 4 ], with well-defined differences in the clinical trajectories seen in these patients compared to adults [ 5 , 6 ]. In recent years, machine learning (ML) has been increasingly employed to improve risk prediction for sepsis, with variable success [ 4 , 5 ].…”

“…However, a persistent issue with ML-sepsis prediction is the problem of data drift, where the eventual application population has different statistical distributions than the training population/data, and the inevitable performance of these systems over time. While retraining has been proposed as a maintenance strategy for these ML systems after deployment [ 6 ], the need to do retraining intrinsically limits the utility of such systems in a mission-critical intensive care setting. With respect to pediatric sepsis, the fact that there are fewer extensive data sets of these patients, compared to adult sepsis, accentuates the limitations of ML predictive algorithms, being more subject to brittleness, overfitting, and a failure to generalize.…”