Evaluation of Domain Generalization and Adaptation on Improving Model Robustness to Temporal Dataset Shift in Clinical Medicine

Abstract

Importance: Temporal dataset shift associated with changes in healthcare over time is a barrier to deploying machine learning-based clinical decision support systems. Algorithms that learn robust models by estimating invariant properties across time periods for domain generalization (DG) and unsupervised domain adaptation (UDA) might be suitable to proactively mitigate dataset shift. Objective: To characterize the impact of temporal dataset shift on clinical prediction models and benchmark DG and UDA algorithms on improving model robustness. Design, Setting, and Participants: In this cohort study, intensive care unit patients from the MIMIC-IV database were categorized by year groups (2008-2010, 2011-2013, 2014-2016 and 2017-2019). Tasks were predicting mortality, long length of stay, sepsis and invasive ventilation. Feedforward neural networks were used as prediction models. The baseline experiment trained models using empirical risk minimization (ERM) on 2008-2010 (ERM[08-10]) and evaluated them on subsequent year groups. DG experiment trained models using algorithms that estimated invariant properties using 2008-2016 and evaluated them on 2017-2019. UDA experiment leveraged unlabelled samples from 2017-2019 for unsupervised distribution matching. DG and UDA models were compared to ERM[08-16] models trained using 2008-2016. Main Outcome(s) and Measure(s): Main performance measures were area-under-the-receiver-operating-characteristic curve (AUROC), area-under-the-precision-recall curve and absolute calibration error. Threshold-based metrics including false-positives and false-negatives were used to assess the clinical impact of temporal dataset shift and its mitigation strategies. Results: In the baseline experiments, dataset shift was most evident for sepsis prediction (maximum AUROC drop, 0.090; 95% confidence interval (CI), 0.080-0.101). Considering a scenario of 100 consecutively admitted patients showed that ERM[08-10] applied to 2017-2019 was associated with one additional false-negative among 11 patients with sepsis, when compared to the model applied to 2008-2010. When compared with ERM[08-16], DG and UDA experiments failed to produce more robust models (range of AUROC difference, -0.003-0.050). Conclusions and Relevance: DG and UDA failed to produce more robust models compared to ERM in the setting of temporal dataset shift. Alternate approaches are required to preserve model performance over time in clinical medicine.

Evaluation of Feature Selection Methods for Preserving Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine

“…Our descriptions of temporal dataset shift on model performance in MIMIC-IV were consistent with previously seen results. 20 There have been few published reports demonstrating methods that can proactively improve model robustness to temporal dataset shift in clinical medicine, 2 21 and recent attempts using domain generalization and unsupervised adaptation were unsuccessful. 20 Previous work exploring causal and noncausal feature selection on medical datasets found that while causal feature selection could mitigate nontemporal covariate shift to an extent, it generally did not outperform regularization-based feature selection.…”

“…20 There have been few published reports demonstrating methods that can proactively improve model robustness to temporal dataset shift in clinical medicine, 2 21 and recent attempts using domain generalization and unsupervised adaptation were unsuccessful. 20 Previous work exploring causal and noncausal feature selection on medical datasets found that while causal feature selection could mitigate nontemporal covariate shift to an extent, it generally did not outperform regularization-based feature selection. 22 While one study showed that causal feature selection outperformed other feature selection methods to mitigate temporal dataset shift, the setting was stock market prediction, a nonclinical setting.…”

Mitigating the Influence of Domain Shift in Skin Lesion Classification: A Benchmark Study of Unsupervised Domain Adaptation Methods on Dermoscopic Images

“…As such domain shifts lead to a drop in performance, there exist different approaches to handle that problem, e.g. data augmentation Yao et al [2022], domain generalization Wang et al [2021] and domain adaptation (DA) Wang and Deng [2018], Guo et al [2021]. Domain generalization and DA are closely related.…”