Evaluating Progress on Machine Learning for Longitudinal Electronic Healthcare Data

Bellamy, David; Celi, Leo; Beam, Andrew L.

doi:10.48550/arxiv.2010.01149

Cited by 6 publications

(7 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To facilitate autoregressive model training, we introduce causal masking on the SGU weight matrix, W , to prevent future time-steps from leaking information. This is efficiently done by zeroing all upper-triangular elements of the matrix before the multiplication in (6).…”

Section: B Components Of the Sansformer Modelmentioning

confidence: 99%

“…In instances where EHRs encompass natural language input, such as discharge notes written by healthcare professionals, Transformers have achieved impressive results [69], [30]. Nonetheless, the unique structure and characteristics of EHR data, especially sequences of clinical codes, present challenges that can sometimes hinder Transformers from consistently outperforming simpler models with carefully engineered features [1], [6], [32]. While Transformers excel in many areas, their performance with EHR can be nuanced, often requiring extensive pretraining on large datasets.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Self-Supervised Forecasting in Electronic Health Records With Attention-Free Models

Kumar,

Ilin,

Salo

et al. 2024

IEEE Trans. Artif. Intell.

View full text Add to dashboard Cite

Despite the proven effectiveness of Transformer neural networks across multiple domains, their performance with Electronic Health Records (EHR) can be nuanced. The unique, multidimensional sequential nature of EHR data can sometimes make even simple linear models with carefully engineered features more competitive. Thus, the advantages of Transformers, such as efficient transfer learning and improved scalability are not always fully exploited in EHR applications.In this work, we aim to forecast the demand for healthcare services, by predicting the number of patient visits to healthcare facilities. The challenge amplifies when dealing with divergent patient subgroups, like those with rare diseases, which are characterized by unique health trajectories and are typically smaller in size. To address this, we employ a self-supervised pretraining strategy, Generative Summary Pretraining (GSP), which predicts future summary statistics based on past health records of a patient. Our models are pretrained on a health registry of nearly one million patients, then fine-tuned for specific subgroup prediction tasks, showcasing the potential to handle the multifaceted nature of EHR data.In evaluation, SANSformer consistently surpasses robust EHR baselines, with our GSP pretraining method notably amplifying model performance, particularly within smaller patient subgroups. Our results illuminate the promising potential of tailored attention-free models and self-supervised pretraining in refining healthcare utilization predictions across various patient demographics.Impact Statement-Large neural networks have demonstrated success in various predictive tasks using Electronic Health Records (EHR). However, their performance in small divergent patient cohorts, such as those with rare diseases, often falls short of simpler linear models due to the substantial data requirements of large models. To address this limitation, we introduce the SANSformers architecture, specifically designed for forecasting healthcare utilization within EHR. Distinct from traditional transformers, SANSformers utilize attention-free mechanisms, thereby reducing complexity. We also present Generative Summary Pretraining (GSP), a self-supervised learning technique that enables large neural networks to maintain predictive efficiency even with smaller patient subgroups. Through extensive evaluation across two real-world datasets, we provide a comparative analysis with existing state-of-the-art EHR prediction models, offering a new perspective on predicting healthcare utilization.

show abstract

Section: B Components Of the Sansformer Modelmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Self-Supervised Forecasting in Electronic Health Records With Attention-Free Models

Kumar,

Ilin,

Salo

et al. 2024

IEEE Trans. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…3C right). This may be due to the fact that specific gene mutations often have an important impact on targeted therapy 27 , but mutation information is difficult to reveal from gene expression data; while chemotherapy drugs were widely reported to be related to gene expression 31,32 so their IC50 is easier to predict. Gene expression-based results had an overall lower PCC, and we did not observe a significant difference between drug types.…”

Section: Scfoundation Improves Cancer Drug Response Predictionmentioning

confidence: 99%

Large Scale Foundation Model on Single-cell Transcriptomics

Hao

Gong²,

Zeng³

et al. 2023

Preprint

View full text Add to dashboard Cite

Large-scale pretrained models have become foundation models, leading to breakthroughs in natural language processing and related fields. Developing foundation models in life science, aimed at deciphering the "languages" of cells and facilitating biomedical research, is challenging yet promising. We developed a large-scale pretrained model, scFoundation, for this purpose. scFoundation was trained on over 50 million human single-cell transcriptomics data, which contain high-throughput observations on the complex molecular features in all known types of cells. scFoundation is currently the largest model in terms of the size of trainable parameters, dimensionality of genes and the number of cells used in the pre-training. Experiments showed that scFoundation can serve as a foundation model for single-cell transcriptomics and can achieve state-of-the-art performances in a diverse array of downstream tasks, such as gene expression enhancement, tissue drug response prediction, single-cell drug response classification, and single-cell perturbation prediction.

show abstract

“…When under-powered baselines are chosen, the algorithm results may seem more favorable, creating an illusion of progress. This is illustrated in healthcare data [Bellamy et al, 2020], semi-supervised learning [Oliver et al, 2018], recommendation systems [Dacrema et al, 2019], metric learning [Musgrave et al, 2020] and deep learning more generally [Marcus, 2018] (and references therein).…”

Section: Incorrectly Chosen Baselinesmentioning

confidence: 99%

How I failed machine learning in medical imaging -- shortcomings and recommendations

Varoquaux,

Cheplygina

2021

Preprint

View full text Add to dashboard Cite

Medical imaging is an important research field with many opportunities for improving patients' health. However, there are a number of challenges that are slowing down the progress of the field as a whole, such optimizing for publication. In this paper we reviewed several problems related to choosing datasets, methods, evaluation metrics, and publication strategies. With a review of literature and our own analysis, we show that at every step, potential biases can creep in. On a positive note, we also see that initiatives to counteract these problems are already being started. Finally we provide a broad range of recommendations on how to further these address problems in the future. For reproducibility, data and code for our analyses are available on https://github.com/GaelVaroquaux/ml med imaging failures.

show abstract

Evaluating Progress on Machine Learning for Longitudinal Electronic Healthcare Data

Cited by 6 publications

References 7 publications

Self-Supervised Forecasting in Electronic Health Records With Attention-Free Models

Self-Supervised Forecasting in Electronic Health Records With Attention-Free Models

Large Scale Foundation Model on Single-cell Transcriptomics

How I failed machine learning in medical imaging -- shortcomings and recommendations

Contact Info

Product

Resources

About