Evaluation of Feature Selection Methods for Preserving Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine

Abstract

Background Temporal dataset shift can cause degradation in model performance as discrepancies between training and deployment data grow over time. The primary objective was to determine whether parsimonious models produced by specific feature selection methods are more robust to temporal dataset shift as measured by out-of-distribution (OOD) performance, while maintaining in-distribution (ID) performance. Methods Our dataset consisted of intensive care unit patients from MIMIC-IV categorized by year groups (2008–2010, 2011–2013, 2014–2016, and 2017–2019). We trained baseline models using L2-regularized logistic regression on 2008–2010 to predict in-hospital mortality, long length of stay (LOS), sepsis, and invasive ventilation in all year groups. We evaluated three feature selection methods: L1-regularized logistic regression (L1), Remove and Retrain (ROAR), and causal feature selection. We assessed whether a feature selection method could maintain ID performance (2008–2010) and improve OOD performance (2017–2019). We also assessed whether parsimonious models retrained on OOD data performed as well as oracle models trained on all features in the OOD year group. Results The baseline model showed significantly worse OOD performance with the long LOS and sepsis tasks when compared with the ID performance. L1 and ROAR retained 3.7 to 12.6% of all features, whereas causal feature selection generally retained fewer features. Models produced by L1 and ROAR exhibited similar ID and OOD performance as the baseline models. The retraining of these models on 2017–2019 data using features selected from training on 2008–2010 data generally reached parity with oracle models trained directly on 2017–2019 data using all available features. Causal feature selection led to heterogeneous results with the superset maintaining ID performance while improving OOD calibration only on the long LOS task. Conclusions While model retraining can mitigate the impact of temporal dataset shift on parsimonious models produced by L1 and ROAR, new methods are required to proactively improve temporal robustness.

Deep continual multitask out-of-hospital incident severity assessment from changing clinical features

“…While there are studies addressing the training and deployment of Machine Learning models in the context of medical data with temporal shifts [11,13,14,[17][18][19], it is challenging to find similar studies for out-of-hospital emergencies. Our approach is centered on changing clinical features instead of free text features as in [11].…”

“…Finally, while there are studies addressing the training and deployment of Machine Learning models in the context of medical data with temporal shifts [12,13,[43][44][45], it is challenging to find similar studies for out-of-hospital emergencies. Furthermore, although the predefined feature domain strategy shares some similarities with the domain invariant feature approach proposed by [12] and the foundational model strategy described in [44], our approach in this work differs from previous solutions.…”

“…Furthermore, although the predefined feature domain strategy shares some similarities with the domain invariant feature approach proposed by [12] and the foundational model strategy described in [44], our approach in this work differs from previous solutions. We do not rely on raw feature aggregation [43], pre-shift patient weighting [13], or parsimonious models [45]. We instead integrate various strategies to address both the variability in the feature domain and the dynamic nature of parameter updating.…”

“…After their design, we have implemented and applied them to multiple temporal sets of out-of-hospital emergency medical data from the Valencian region, to assess their effectiveness. Despite the existence of previous works addressing the training and deployment of Machine Learning models in the context of medical data with temporal dataset shifts [11,13,14,[17][18][19], to our knowledge, this is the first study facing evolving clinical features within the domain of out-of-hospital emergencies.…”

Deeep Continual Multimodal Multitask Modells for Out-of-Hospital Emergency Medical Call Incidents Triage Support in the Presence of Dataset Shifts

“…These systems have the potential to positively impact patient well-being and enhance the sustainability of health services. Although we can find studies concerned with the development Machine Learning models focused on dealing with medical data in the presence of temporal distributional drifts (Guo et al, 2023), (Guo et al, 2022), (Lemmon et al, 2023), to the best of our knowledge, this is the first study to tackle real EMCI data using a Continual Learning approach, representing a significant contribution to the field and one of the earliest real-world applications of Continual Learning methods.…”