Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Data

Abstract

Widespread adoption of electronic health records (EHRs) has fueled development of clinical outcome models using machine learning. However, patient EHR data are complex, and how to optimally represent them is an open question. This complexity, along with often small training set sizes available to train these clinical outcome models, are two core challenges for training high quality models. In this paper, we demonstrate that learning generic representations from the data of all the patients in the EHR enables better performing prediction models for clinical outcomes, allowing for these challenges to be overcome. We adapt common representation learning techniques used in other domains and find that representations inspired by language models enable a 3.5% mean improvement in AUROC on five clinical outcomes compared to standard baselines, with the average improvement rising to 19% when only a small number of patients are available for training a prediction model for a given clinical outcome.

Deep Contextual Clinical Prediction with Reverse Distillation

“…Many recent works analyze how deep learning can be applied to clinical prediction (Choi et al 2016a;Rajkomar et al 2018;Che et al 2018;Steinberg et al 2020;Choi et al 2016b;Harutyunyan et al 2019;Gao et al 2020;Ma et al 2018;Zhang et al 2019). Several approaches use recurrent neural networks (RNNs) to ingest medical records, and achieve excellent performance on tasks like predicting in-patient mortality upon hospital admission (Choi et al 2016a).…”

MedGPT: Medical Concept Prediction from Clinical Narratives

“…The model is evaluated on a small subset of disorders which may be insufficient for estimating general performance. Apart from BERT based models, we also note Long Short Term Memory (LSTM) models, like the one proposed by Ethan Steinberg et al [20]. Similar to the other models, they only use structured data and fine-tune their model for the prediction of limited future events.…”