Prospective validation of a machine learning model that uses provider notes to identify candidates for resective epilepsy surgery

Abstract

Objective: Delay to resective epilepsy surgery results in avoidable disease burden and increased risk of mortality. The objective was to prospectively validate a natural language processing (NLP) application that uses provider notes to assign epilepsy surgery candidacy scores. Methods: The application was trained on notes from (1) patients with a diagnosis of epilepsy and a history of resective epilepsy surgery and (2) patients who were seizure-free without surgery. The testing set included all patients with unknown surgical candidacy status and an upcoming neurology visit. Training and testing sets were updated weekly for 1 year. One-to three-word phrases contained in patients' notes were used as features. Patients prospectively identified by the application as candidates for surgery were manually reviewed by two epileptologists. Performance metrics were defined by comparing NLP-derived surgical candidacy scores with surgical candidacy status from expert chart review. Results: The training set was updated weekly and included notes from a mean of 519 ± 67 patients. The area under the receiver operating characteristic curve (AUC) from 10-fold cross-validation was 0.90 ± 0.04 (range = 0.83-0.96) and improved by 0.002 per week (P < .001) as new patients were added to the training set. Of the 6395 patients who visited the neurology clinic, 4211 (67%) were evaluated by the model.The prospective AUC on this test set was 0.79 (95% confidence interval [CI] = 0.62-0.96). Using the optimal surgical candidacy score threshold, sensitivity was 0.80 (95% CI = 0.29-0.99), specificity was 0.77 (95% CI = 0.64-0.88), positive predictive value was 0.25 (95% CI = 0.07-0.52), and negative predictive value was 0.98 (95% CI = 0.87-1.00). The number needed to screen was 5.6. Significance: An electronic health record-integrated NLP application can accurately assign surgical candidacy scores to patients in a clinical setting. K E Y W O R D Sclinical decision support, epilepsy surgery, machine learning, natural language processing 40 | WISSEL Et aL.

Early identification of epilepsy surgery candidates: A multicenter, machine learning study

“…Machine learning methodologies (ML) can be used to identify candidates for epilepsy surgery years before they undergo surgery. [12][13][14][15] ML algorithms are infrequently implemented into care. In pediatrics, one algorithm was fully automated including the provision of decision support to providers 13 ; however, the positive predictive value (PPV) of the alerts from this system was low at only 25%.…”

Identification of focal epilepsy by diffusion tensor imaging using machine learning

“…[7][8][9] There is a huge demand to extend the application of machine learning in the epilepsy field, such as diagnosis of epilepsy, localization of epileptogenic lesions, and medical or surgical management. 7,[10][11][12][13][14][15][16][17][18][19][20] Studies have investigated the feasibility of machine learning to distinguish patients with epilepsy versus healthy controls based on a variety of data sources, such as functional MRI, 18 DTI, 15,19 or magnetic resonance encephalography. 20 However, these studies are limited by modest sample sizes.…”

Can antiepileptic drug efficacy be studied from electronic health records? A review of current approaches

“…Wissel et al used n-grams (up to n=3) and SVM to achieve sensitivity 80%, specificity 77%, PPV 25%, and NPV 98% for determining medically refractory epilepsy. [38]…”