Development and validation of a prediction model for actionable aspects of frailty in the text of clinicians’ encounter notes

Abstract

Objective Frailty is a prevalent risk factor for adverse outcomes among patients with chronic lung disease. However, identifying frail patients who may benefit from interventions is challenging using standard data sources. We therefore sought to identify phrases in clinical notes in the electronic health record (EHR) that describe actionable frailty syndromes. Materials and Methods We used an active learning strategy to select notes from the EHR and annotated each sentence for 4 actionable aspects of frailty: respiratory impairment, musculoskeletal problems, fall risk, and nutritional deficiencies. We compared the performance of regression, tree-based, and neural network models to predict the labels for each sentence. We evaluated performance with the scaled Brier score (SBS), where 1 is perfect and 0 is uninformative, and the positive predictive value (PPV). Results We manually annotated 155 952 sentences from 326 patients. Elastic net regression had the best performance across all 4 frailty aspects (SBS 0.52, 95% confidence interval [CI] 0.49–0.54) followed by random forests (SBS 0.49, 95% CI 0.47–0.51), and multi-task neural networks (SBS 0.39, 95% CI 0.37–0.42). For the elastic net model, the PPV for identifying the presence of respiratory impairment was 54.8% (95% CI 53.3%–56.6%) at a sensitivity of 80%. Discussion Classification models using EHR notes can effectively identify actionable aspects of frailty among patients living with chronic lung disease. Regression performed better than random forest and neural network models. Conclusions NLP-based models offer promising support to population health management programs that seek to identify and refer community-dwelling patients with frailty for evidence-based interventions.

Machine Learning Approaches for Electronic Health Records Phenotyping: A Methodical Review

“…Overall, application papers are the most common type and primarily utilize supervised learning methods. Applications developed phenotyping algorithms for diverse uses, including development of clinical decision support tools [51,93,95] and phenome-wide association studies [66,68]. Among methods articles, approaches to alleviate the burden of labeling, such as semi-supervised and weakly-supervised learning, are the most prevalent.…”

“…feature engineering or labeling). Within the articles, corpora included notes from EHRs [33,83,[90][91][92] as well as external sources such as biomedical publications [34,72,[93][94][95][96] and articles from Wikipedia and Google News [32,69,97].…”

“…Among the methods used to train word embeddings, Word2vec [98] and Bidirectional Encoder Representations from Transformers (BERT) and variants [99][100][101][102] are the most frequently used (Supplementary Material Table S9). Word embeddings typically serve as the input layer to phenotyping algorithms using deep learning models and have also been used to account for ambiguous abbreviations and spelling errors in clinical notes [25,27,29,32,34,42,69,72,[74][75][76][77]83,[93][94][95][96][97][103][104][105][106][107][108][109][110][111][112][113][114].…”

“…33 of the 72 supervised learning articles employ deep learning, with 31 of these articles using word embeddings as the input layer. Feed-forward neural networks are the least frequently used architecture [25,32,108,111,118,133], while recurrent neural networks (RNNS), [27,34,59,69,[74][75][76]85,90,94,95,103,[105][106][107]109,110,123,126,130,134], convolutional neural networks (CNNs) [33,75,83,91,93,94,103,106,107,112,126,[134][135][136], and their variants are frequently used (Supplementary Material Table S12). RNNs are particularly well suited for sequential data from longitudinal EHRs.…”

“…In spite of the advantages of deep learning, traditional supervised learning methods have outperformed their deep learning counterparts. For example, Martin et al found that a penalized elastic net logistic regression consistently outperformed FFNNs for identifying frailty aspects such as musculoskeletal problems and respiratory impairment (F1-score Elastic net 0.51-0.65, F1-score FFNN 0.24-0.62) [95]. Several authors also commented on challenges related to model interpretability and transportability of deep learning models across sites [25,85,91,130].…”