Natural language processing of Veterans’ electronic health records to confirm diagnoses of monoclonal gammopathy of undetermined significance.

Abstract

1557 Background: The Veterans Health Administration (VHA) provides extensive electronic health records (EHRs) on Veterans nationwide. Our prior studies utilized VHA data to study the risk of progression from monoclonal gammopathy of undetermined significance (MGUS) to multiple myeloma. These studies relied on International Classification of Disease (ICD) codes and manual abstraction on clinical notes to both identify and verify MGUS patients. Diagnosis confirmation is necessary because many providers place a diagnosis on the clinical notes to order lab tests, which is often left in the EHR despite a negative test result. However, manual abstraction is labor intensive and time consuming. With the advancement in natural language processing (NLP), we developed a model to make MGUS confirmation more efficient. Methods: We randomly selected 700 patients within patients diagnosed with MGUS from 1999-2021 in the VHA identified via ICD codes. A random sample of 500 patients were selected and split into the training (80%) and the testing (20%) sets. The remainder (n = 200) served as the validation set. There were 32,708 unstructured hematology/oncology Text Integration Utility reports and 9,237 lab reports (including 2,322 discrete results and 6,915 unstructured comments). All reports were manually reviewed to confirm MGUS diagnoses and served as the reference standard. We compiled three lists of keywords suggestive of MGUS diagnosis, subtypes of immunoglobulins, and negation modifiers. We trained a symbolic NLP model to identify diagnoses using combinations of the lists along with M-protein levels from lab reports. The optimized combination that gave the highest recall and precision from the training set was used and evaluated on the testing and validation sets. Results: Among patients with ICD codes for MGUS, manual abstraction confirmed 84% MGUS diagnoses in the testing set and 80% in the validation set. Our NLP model in the training set confirmed 75% and achieved recall, precision, accuracy, and F1 score of 88.1, 98.7, 89.0, and 93.1%, respectively; in the validation set, our rule confirmed 76% patients and the recall, precision, accuracy, and F1 score were 89.4, 94.7, 87.5, and 92.0%, respectively. On average data abstraction took five minutes per patient (excluding data loading time), whereas NLP model completed 13 patients per minute. Conclusions: The developed NLP model to confirm MGUS diagnosis improves accuracy in diagnosis, compared to ICD codes alone. While the performance is similar to that of manual abstraction, our NLP model is an efficient and viable method in MGUS diagnosis confirmation. [Table: see text]

The Association of Agent Orange (AO) Exposure with Monoclonal Gammopathy of Undetermined Significance (MGUS) to Multiple Myeloma (MM) Progression: A Population-based Study of Vietnam War Era Veterans

“…Third, we utilized a highly accurate NLP algorithm to con rm that clinical notes and diagnostic data support the MM diagnoses. [19][20][21][22] We adjusted for known risk factors for MGUS progression including M-protein level, MGUS subtype, and obesity and for the competing risk of death in the analyses. At present, MGUS is managed by annual every 2-3 year follow up with SPEP and/or serum free light chains.…”

“…This step is crucial as our prior studies utilizing data abstraction demonstrated that, at best, only 80–84% of patients with ICD codes for MGUS were confirmed to have MGUS on manual chart review. 20 …”

“…We identi ed patients diagnosed with MGUS from 10/1/1999-12/31/2021 via ICD-10 code, 47.2, then con rmed their diagnosis status and dates by an established natural language processing (NLP) algorithm. [19][20][21][22] Patients were considered to have MGUS if they had M-protein detected on serum protein electrophoresis (SPEP) with immuno xation con rmation. This step is crucial as our prior studies utilizing data abstraction demonstrated that, at best, only 80-84% of patients with ICD codes for MGUS were con rmed to have MGUS on manual chart review.…”

“…This step is crucial as our prior studies utilizing data abstraction demonstrated that, at best, only 80-84% of patients with ICD codes for MGUS were con rmed to have MGUS on manual chart review. 20 Institutional Review Boards at both Washington University School of Medicine and Veteran Affairs Saint Louis Healthcare System approved the study.…”

“…The progression status and dates were detected by identifying patients in the analytic cohort who had an ICD-10 code for MM, C90.0, and also received MM-speci c treatment (see Supplemental Table 1). Progression was then further con rmed by a published NLP algorithm 19,20 by capturing mentioning of MM diagnosis in the clinical notes and bone marrow biopsies (BMBx) with 10% or more plasma cells and/or SPEP demonstrating 3 g/dL or more M-protein. 28 These con rmation steps were included due to prior studies demonstrating that ICD codes for MM may be only 58% accurate in capturing true MM.…”