Self-Supervised Forecasting in Electronic Health Records With Attention-Free Models

Abstract

Despite the proven effectiveness of Transformer neural networks across multiple domains, their performance with Electronic Health Records (EHR) can be nuanced. The unique, multidimensional sequential nature of EHR data can sometimes make even simple linear models with carefully engineered features more competitive. Thus, the advantages of Transformers, such as efficient transfer learning and improved scalability are not always fully exploited in EHR applications.In this work, we aim to forecast the demand for healthcare services, by predicting the number of patient visits to healthcare facilities. The challenge amplifies when dealing with divergent patient subgroups, like those with rare diseases, which are characterized by unique health trajectories and are typically smaller in size. To address this, we employ a self-supervised pretraining strategy, Generative Summary Pretraining (GSP), which predicts future summary statistics based on past health records of a patient. Our models are pretrained on a health registry of nearly one million patients, then fine-tuned for specific subgroup prediction tasks, showcasing the potential to handle the multifaceted nature of EHR data.In evaluation, SANSformer consistently surpasses robust EHR baselines, with our GSP pretraining method notably amplifying model performance, particularly within smaller patient subgroups. Our results illuminate the promising potential of tailored attention-free models and self-supervised pretraining in refining healthcare utilization predictions across various patient demographics.Impact Statement-Large neural networks have demonstrated success in various predictive tasks using Electronic Health Records (EHR). However, their performance in small divergent patient cohorts, such as those with rare diseases, often falls short of simpler linear models due to the substantial data requirements of large models. To address this limitation, we introduce the SANSformers architecture, specifically designed for forecasting healthcare utilization within EHR. Distinct from traditional transformers, SANSformers utilize attention-free mechanisms, thereby reducing complexity. We also present Generative Summary Pretraining (GSP), a self-supervised learning technique that enables large neural networks to maintain predictive efficiency even with smaller patient subgroups. Through extensive evaluation across two real-world datasets, we provide a comparative analysis with existing state-of-the-art EHR prediction models, offering a new perspective on predicting healthcare utilization.

Deep learning-based prediction of one-year mortality in Finland is an accurate but unfair aging marker

“…Third, we did not consider other DL model architectures beyond RNNs; however, we used simpler models, that is, penalized logistic regression and XGBoost. Previous work showed that RNNs have comparable performance to other sequential DL models in predicting clinical events 29,30 . More work is needed to identify models for aging clocks that can balance interpretability, fairness, scalability and prediction performance.…”

“…1d). RNNs are effective in modeling patients' health histories 28 and have demonstrated comparable performance to other sequential DL models, such as transformers, in predicting clinical events 29,30 .…”

“…These complexities resemble the challenges also faced in natural language processing as individual life events resemble individual words in natural language. Thus, we used an RNN, namely a gated recurrent unit, which performed similarly or better than a transformer and other commonly used models with sequential DL architecture for clinical event predictions 28,29,49 . Longitudinally expressed records after embedding individuals' lives year by year were used as inputs to a recurrent layer (Fig.…”