Use of Natural Language Processing of Patient-Initiated Electronic Health Record Messages to Identify Patients With COVID-19 Infection

Abstract

ImportanceNatural language processing (NLP) has the potential to enable faster treatment access by reducing clinician response time and improving electronic health record (EHR) efficiency.ObjectiveTo develop an NLP model that can accurately classify patient-initiated EHR messages and triage COVID-19 cases to reduce clinician response time and improve access to antiviral treatment.Design, Setting, and ParticipantsThis retrospective cohort study assessed development of a novel NLP framework to classify patient-initiated EHR messages and subsequently evaluate the model’s accuracy. Included patients sent messages via the EHR patient portal from 5 Atlanta, Georgia, hospitals between March 30 and September 1, 2022. Assessment of the model’s accuracy consisted of manual review of message contents to confirm the classification label by a team of physicians, nurses, and medical students, followed by retrospective propensity score–matched clinical outcomes analysis.ExposurePrescription of antiviral treatment for COVID-19.Main Outcomes and MeasuresThe 2 primary outcomes were (1) physician-validated evaluation of the NLP model’s message classification accuracy and (2) analysis of the model’s potential clinical effect via increased patient access to treatment. The model classified messages into COVID-19–other (pertaining to COVID-19 but not reporting a positive test), COVID-19-positive (reporting a positive at-home COVID-19 test result), and non–COVID-19 (not pertaining to COVID-19).ResultsAmong 10 172 patients whose messages were included in analyses, the mean (SD) age was 58 (17) years; 6509 patients (64.0%) were women and 3663 (36.0%) were men. In terms of race and ethnicity, 2544 patients (25.0%) were African American or Black, 20 (0.2%) were American Indian or Alaska Native, 1508 (14.8%) were Asian, 28 (0.3%) were Native Hawaiian or other Pacific Islander, 5980 (58.8%) were White, 91 (0.9%) were more than 1 race or ethnicity, and 1 (0.01%) chose not to answer. The NLP model had high accuracy and sensitivity, with a macro F1 score of 94% and sensitivity of 85% for COVID-19–other, 96% for COVID-19–positive, and 100% for non–COVID-19 messages. Among the 3048 patient-generated messages reporting positive SARS-CoV-2 test results, 2982 (97.8%) were not documented in structured EHR data. Mean (SD) message response time for COVID-19–positive patients who received treatment (364.10 [784.47] minutes) was faster than for those who did not (490.38 [1132.14] minutes; P = .03). Likelihood of antiviral prescription was inversely correlated with message response time (odds ratio, 0.99 [95% CI, 0.98-1.00]; P = .003).Conclusions and RelevanceIn this cohort study of 2982 COVID-19–positive patients, a novel NLP model classified patient-initiated EHR messages reporting positive COVID-19 test results with high sensitivity. Furthermore, when responses to patient messages occurred faster, patients were more likely to receive antiviral medical prescription within the 5-day treatment window. Although additional analysis on the effect on clinical outcomes is needed, these findings represent a possible use case for integration of NLP algorithms into clinical care.

Utilizing Natural Language Processing and Large Language Models in the Diagnosis and Prediction of Infectious Diseases: A Systematic Review

“…After the removal of 143 duplicates, the screening process found 13 studies that met our inclusion and exclusion criteria (19,(21)(22)(23)(24)(25)(26)(27)(28)(29)(30)(31)(32). We identified two additional studies via reference screening (33,34).…”

“…The eCOV model was used for COVID-19 electronic health records (EHR) messages (32). eCOV’s classification scored a 94% macro F1.…”

“…It is not a one-size-fits-all solution but requires tailored approaches depending on the disease context. Secondly, the use of AI in public health surveillance, as evidenced by the BERTweet model for Lyme Disease and the eCOV model for COVID-19, is a breakthrough (19,24,32). These tools demonstrate how AI can transform unstructured data from diverse sources into actionable insights, a critical advancement for early outbreak detection and response.…”

“…Six studies demonstrated a low risk of bias across all domains (21,(31)(32)(33)(34)(35). Overall, ten studies were evaluated as having low risk (21)(22)(23)26,28,(31)(32)(33)(34)(35), four showed high risk (19,24,27,29), and one lacked sufficient data for a conclusive risk assessment (30).…”

Clinician-Trained Artificial Intelligence for Enhanced Routing of Patient Portal Messages in the Electronic Health Record

“…We have recently demonstrated the feasibility of using NLP to automatically prioritize urgent patient messages and identify public health trends from the EHR. 14 Although additional studies have also shown the potential of using text search and NLP to analyze and label messages effectively, [15][16][17][18][19] none have explored the value of adding machine learning (ML) to automate message triage through intelligent rerouting. 20 No previously published efforts have used high-level NLP to analyze and intelligently reroute patient messages.…”