Hi-BEHRT: Hierarchical Transformer-Based Model for Accurate Prediction of Clinical Events Using Multimodal Longitudinal Electronic Health Records

Abstract

Electronic health records (EHR) represent a holistic overview of patients' trajectories. Their increasing availability has fueled new hopes to leverage them and develop accurate risk prediction models for a wide range of diseases. Given the complex interrelationships of medical records and patient outcomes, deep learning models have shown clear merits in achieving this goal. However, a key limitation of current study remains their capacity in processing long sequences, and long sequence modelling and its application in the context of healthcare and EHR remains unexplored. Capturing the whole history of medical encounters is expected to lead to more accurate predictions, but the inclusion of records collected for decades and from multiple resources can inevitably exceed the receptive field of the most existing deep learning architectures. This can result in missing crucial, long-term dependencies. To address this gap, we present Hi-BEHRT, a hierarchical Transformer-based model that can significantly expand the receptive field of Transformers and extract associations from much longer sequences. Using a multimodal largescale linked longitudinal EHR, the Hi-BEHRT exceeds the state-of-the-art deep learning models 1% to 5% for area under the receiver operating characteristic (AUROC) curve and 1% to 8% for area under the precision recall (AUPRC) curve on average, and 2% to 8% (AUROC) and 2% to 11% (AUPRC) for patients with long medical history for 5-year heart failure, diabetes, chronic kidney disease, and stroke risk prediction. Additionally, because pretraining for Manuscript

Inter operator variability of machine learning researchers predicting all-cause mortality in patients admitted to intensive care unit

“…When building machine learning models for healthcare applications, algorithms need to be compared to appropriate baselines or standards to ensure the performance of these models is high enough to warrant the cost of deploying and maintaining them [26]. Performance of models is typically evaluated in relation to other outcomes [35], [36], [37], [38], [39], [40] or other prediction models [35], [41], [42], [43], [32], [44], [45]. To this end, we test the models built in this study which predict sudden death and other catastrophic cardiovascular events in two ways.…”

Soft-prompt tuning to predict lung cancer using primary care free-text Dutch medical notes

“…Recent years have seen the release of many English-language PLMs (pre)trained on clinical data [5,20,12,9]. Dutch-language PLMs are much more rare, and even more so in the medical domain: we are only aware of Verkijk and Vossen's PLM [21], which is pretrained on hospital patient notes.…”

EHR-ML: A generalisable pipeline for reproducible clinical outcomes using electronic health records

“…To establish standardization in this process, various frameworks[23, 24] and guidelines [25, 26, 27, 28, 29] are proposed for conducting and reporting such studies. Additionally, generic toolkits for predictive modeling have been developed to accelerate AI adoption in healthcare[30, 31, 32, 6]. However, these toolkits often suffer from some significant limitations.…”

“…However, these toolkits often suffer from some significant limitations. Restricting the data source to a single rigid input format [31, 32], hinders their applicability, while non-interpretable neural networks raise implementation concerns in healthcare settings [30, 6]. Furthermore, existing solutions lack end-to-end automation, covering data sourcing to model building, posing a roadblock for widespread localized deployments.…”