Stroke Outcome Measurements From Electronic Medical Records: Cross-sectional Study on the Effectiveness of Neural and Nonneural Classifiers

Abstract

Background With the rapid adoption of electronic medical records (EMRs), there is an ever-increasing opportunity to collect data and extract knowledge from EMRs to support patient-centered stroke management. Objective This study aims to compare the effectiveness of state-of-the-art automatic text classification methods in classifying data to support the prediction of clinical patient outcomes and the extraction of patient characteristics from EMRs. Methods Our study addressed the computational problems of information extraction and automatic text classification. We identified essential tasks to be considered in an ischemic stroke value-based program. The 30 selected tasks were classified (manually labeled by specialists) according to the following value agenda: tier 1 (achieved health care status), tier 2 (recovery process), care related (clinical management and risk scores), and baseline characteristics. The analyzed data set was retrospectively extracted from the EMRs of patients with stroke from a private Brazilian hospital between 2018 and 2019. A total of 44,206 sentences from free-text medical records in Portuguese were used to train and develop 10 supervised computational machine learning methods, including state-of-the-art neural and nonneural methods, along with ontological rules. As an experimental protocol, we used a 5-fold cross-validation procedure repeated 6 times, along with subject-wise sampling. A heatmap was used to display comparative result analyses according to the best algorithmic effectiveness (F1 score), supported by statistical significance tests. A feature importance analysis was conducted to provide insights into the results. Results The top-performing models were support vector machines trained with lexical and semantic textual features, showing the importance of dealing with noise in EMR textual representations. The support vector machine models produced statistically superior results in 71% (17/24) of tasks, with an F1 score >80% regarding care-related tasks (patient treatment location, fall risk, thrombolytic therapy, and pressure ulcer risk), the process of recovery (ability to feed orally or ambulate and communicate), health care status achieved (mortality), and baseline characteristics (diabetes, obesity, dyslipidemia, and smoking status). Neural methods were largely outperformed by more traditional nonneural methods, given the characteristics of the data set. Ontological rules were also effective in tasks such as baseline characteristics (alcoholism, atrial fibrillation, and coronary artery disease) and the Rankin scale. The complementarity in effectiveness among models suggests that a combination of models could enhance the results and cover more tasks in the future. Conclusions Advances in information technology capacity are essential for scalability and agility in measuring health status outcomes. This study allowed us to measure effectiveness and identify opportunities for automating the classification of outcomes of specific tasks related to clinical conditions of stroke victims, and thus ultimately assess the possibility of proactively using these machine learning techniques in real-world situations.

Applications of Natural Language Processing for the Management of Stroke Disorders: Scoping Review

“…Among these, BERT has been found to be particularly widely used in the medical field in general, and for stroke in particular, along with specialized versions fitted to these applications that improve their performance [ 22 , 41 ]. More basic ML algorithms and hybrid approaches with rule-based techniques are still more present than advanced DL networks in the recent research on NLP for stroke, and in some cases, tailored rule-based systems outperformed BERT and its derivatives [ 19 , 22 ]. Support vector machine methods were also found to perform better than BERT in one study [ 19 ], although random forest was reported to have the best performance more frequently than any other ML method in the set of reviewed studies [ 18 , 42 , 46 ].…”

“…More basic ML algorithms and hybrid approaches with rule-based techniques are still more present than advanced DL networks in the recent research on NLP for stroke, and in some cases, tailored rule-based systems outperformed BERT and its derivatives [ 19 , 22 ]. Support vector machine methods were also found to perform better than BERT in one study [ 19 ], although random forest was reported to have the best performance more frequently than any other ML method in the set of reviewed studies [ 18 , 42 , 46 ]. Some of these results may seem unexpected, given the remarkable performance of DL in general, and particularly large language models (LLMs), in other areas.…”

“…Table 4 shows the frequency of each category used to classify the answers to the primary and secondary questions, except for the question about the specificity of algorithms and NLP tools for stroke, since there was little variability in those answers. Zhao et al [18] Diagnostic reports CLINICAL: care information management, characterize patients, prediction of outcomes, risk assess-Prognosis (outcomes) Zanotto et al [19] ment; SOCIETAL: supporting research studies; ECONOMIC: public health management Annotated medical images, clinical scales, demographic data, diagnostic reports, medical history, patient treatments CLINICAL: prediction of outcomes Prognosis (outcomes) Sung et al [20] Annotated medical images, clinical scales, demographic data, diagnostic reports, functional outcomes data CLINICAL: prediction of outcomes, risk assessment Prognosis (outcomes) Sung et al [21] Annotated medical images, diagnostic reports CLINICAL: prediction of outcomes, risk assessment Prognosis (outcomes) Miller et al [22] Diagnostic reports CLINICAL: improved triage Diagnosis (classification) Mayampurath et al [23] Demographic data, diagnostic reports, medical history, medication, patient treatments CLINICAL: care information management Prognosis (recurrence) Lineback et al [24] Demographic data, clinical scales, medical history, patient treatments, medication CLINICAL: administration of treatments, care information management, improved triage, prediction of outcomes Prognosis (outcomes) Kogan et al [25] Annotated medical images, diagnostic reports CLINICAL: prediction of outcomes Prognosis (outcomes) Heo et al [26] Annotated medical images, clinical scales, diagnostic reports, medical history CLINICAL: administration of treatments Diagnosis (details); treatment Deng et al [27] Annotated medical images, diagnostic reports, medical history, medication CLINICAL: stroke cause prediction Diagnosis (classification) Bacchi et al [28] Annotated medical images, diagnostic reports CLINICAL: improved triage; ECONOMIC: public health management Diagnosis (details) Yu et al [29] Annotated medical images, diagnostic reports CLINICAL: disease surveillance, improved triage; ECONOMIC: public health management Diagnosis (classification) Wheater et al [30] Diagnostic reports CLINICAL: administration of treatments, care information management, disease surveillance; ECONOMIC: public health management Prevention and diagnosis (classification) Sung et al [31] Diagnostic reports, laboratory test results, medical history CLINICAL: administration of treatments Diagnosis (details); treatment Sung et al [32] Demographic data, medical history CLINICAL: care information management Diagnosis (comorbidities) Shek et al [33] Annotated medical images, diagnostic reports CLINICAL: improved triage Diagnosis (classification) Rannikmäe et al…”

A Machine Learning Approach to Support Urgent Stroke Triage Using Administrative Data and Social Determinants of Health at Hospital Presentation: Retrospective Study

“…Furthermore, various obstacles such as the lack of interoperability have limited the full use of EHR data to improve the delivery of care. Consequently, existing studies are mostly based on patient data confined to a single EHR system within a single geographic area [ 83 ]. By contrast, administrative data, such as claims data, follow specific standards for both the structure and meaning of the variables contained within a claim, and nearly every health care provider must submit electronic claims in the same format to their payers or clearinghouses.…”