Predicting seizure outcome after epilepsy surgery: do we need more complex models, larger samples, or better data?

Abstract

Objective: The accurate prediction of seizure freedom after epilepsy surgery remains challenging. We investigated if 1) training more complex models, 2) recruiting larger sample sizes, or 3) using data-driven selection of clinical predictors would improve our ability to predict post-operative seizure outcome. We also conducted the first external validation of a machine learning model trained to predict post-operative seizure outcome. Methods: We performed a retrospective cohort study of 797 children who had undergone resective or disconnective epilepsy surgery at a single tertiary center. We extracted patient information from medical records and trained three models - a logistic regression, a multilayer perceptron, and an XGBoost model - to predict one-year post-operative seizure outcome on our dataset. We evaluated the performance of a recently published XGBoost model on the same patients. We further investigated the impact of sample size on model performance, using learning curve analysis to estimate performance at samples up to N=2,000. Finally, we examined the impact of predictor selection on model performance. Results: Our logistic regression achieved an accuracy of 72% (95% CI=68-75%, AUC=0.72), while our multilayer perceptron and XGBoost both achieved accuracies of 71% (95% CI-MLP=67-74%, AUC-MLP=0.70; 95% CI-XGBoost own=68-75%, AUC-XGBoost own=0.70). There was no significant difference in performance between our three models (all P>0.4) and they all performed better than the external XGBoost, which achieved an accuracy of 63% (95% CI=59-67%, AUC=0.62; P-LR=0.005, P-MLP=0.01, P-XGBoost own=0.01) on our data. All models showed improved performance with increasing sample size, with limited improvements above N=400. The best model performance was achieved with data-driven feature selection. Significance: We show that neither the deployment of complex machine learning models nor the assembly of thousands of patients alone is likely to generate significant improvements in our ability to predict post-operative seizure freedom. We instead propose that improved feature selection alongside collaboration, data standardization, and model sharing is required to advance the field.

Exit Strategy: Balancing the Risks and Rewards of Antiseizure Medication Withdrawal

“…Complex machine learning models are unlikely to solve the problem, overdetecting noise without addressing data limitations. 33 Better data are needed to evaluate underexplored predictors (eg, specific epilepsy syndromes, genetics, sleep, substances [eg, illicit drugs or alcohol], and more advanced EEG and imaging analysis). Also, additional work should compute not only post-discontinuation risk but also individualized seizure risk increases due to withdrawal and evaluate the effect of withdrawal on different seizure types and frequency beyond dichotomizing as seizure-free versus not. How should we optimally integrate seizure risk calculators into clinical practice and communicate that risk to patients, and what might be their impact on clinical care?…”

“…Future models could utilize larger datasets, apply more precise variable selection, and integrate cutting-edge tools like artificial intelligence, neuroimaging, and genetic profiling to significantly improve model performance, thereby advancing the precision and accuracy of personalized treatment regimens. 33…”

Epilepsy surgery for children with epileptic spasms: A systematic review and meta‐analysis with focus on predictors and outcomes

“…31,32 Unfortunately, despite national and international guidelines advocating for prompt referral, many children with drug-resistant epilepsy still face delayed surgical evaluation. 33 Recent research by Buttle et al examined the attitudes of child neurologists practicing in North America, reporting that 80% of surveyed pediatric neurologists stated that they would be reluctant to refer patients with generalized EEG findings or generalized seizures for surgical evaluation. 34 Although spasm semiology and EEG findings are not always helpful to localize the lesion for patients with epileptic spasms, this should not deter patients from being considered for epilepsy surgery.…”

“…Unfortunately, despite national and international guidelines advocating for prompt referral, many children with drug‐resistant epilepsy still face delayed surgical evaluation 33 . Recent research by Buttle et al.…”

Back to the Basics in Predictive Modeling—Predicting Surgical Success

“…Erikkson et al 3 took a step back and asked which of several common issues most limit a model’s accuracy. They retrospectively extracted medical records from 797 children who received either surgical resection or a disconnection procedure between 2000 and 2018 at the Great Ormond Street Hospital.…”