External Validation of a Neural Network Model in Aneurysmal Subarachnoid Hemorrhage: A Comparison With Conventional Logistic Regression Models

Abstract

BACKGROUND:Interest in machine learning (ML)–based predictive modeling has led to the development of models predicting outcomes after aneurysmal subarachnoid hemorrhage (aSAH), including the Nijmegen acute subarachnoid hemorrhage calculator (Nutshell). Generalizability of such models to external data remains unclear.OBJECTIVE:To externally validate the performance of the Nutshell tool while comparing it with the conventional Subarachnoid Hemorrhage International Trialists (SAHIT) models and to review the ML literature on outcome prediction after aSAH and aneurysm treatment.METHODS:A prospectively maintained database of patients with aSAH presenting consecutively to our institution in the 2013 to 2018 period was used. The web-based Nutshell and SAHIT calculators were used to derive the risks of poor long-term (12-18 months) outcomes and 30-day mortality. Discrimination was evaluated using the area under the curve (AUC), and calibration was investigated using calibration plots. The literature on relevant ML models was surveyed for a synopsis.RESULTS:In 269 patients with aSAH, the SAHIT models outperformed the Nutshell tool (AUC: 0.786 vs 0.689, P = .025) in predicting long-term functional outcomes. A logistic regression model of the Nutshell variables derived from our data achieved adequate discrimination (AUC = 0.759) of poor outcomes. The SAHIT models outperformed the Nutshell tool in predicting 30-day mortality (AUC: 0.810 vs 0.636, P < .001). Calibration properties were more favorable for the SAHIT models. Most published aneurysm-related ML-based outcome models lack external validation and usable testing platforms.CONCLUSION:The Nutshell tool demonstrated limited performance on external validation in comparison with the SAHIT models. External validation and the dissemination of testing platforms for ML models must be emphasized.

Letter to the Editor. Machine learning–based prediction models in neurosurgery

“…Neurosurg Focus. 2023;54 [6]:E8). We congratulate the authors on meticulously assessing, per the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) criteria, 2 an important number of ML prediction models described over the past 3 years in five reputable neurosurgical journals.…”

“…Such tools could assist surgeons and patients in making informed decisions about performing surgery or watchful waiting when a diagnosis is not clear, as with previous attempts at distinguishing radiation necrosis from tumor progression. 5,6 For example, a patient with a high probability of radiation necrosis over tumor recurrence could potentially avoid surgery, its complications, and the associated burden on the patient, family, and healthcare system.…”

“…1,2 We agree that predictive models have the potential to significantly augment medical practice. While the importance of external validation in neurosurgery has been highlighted 3 and some studies on external validation have been published, [4][5][6] these efforts remain rare. We support Habashy and colleagues' call for considering external validation as a component of the initial reporting of a ML prediction model to address concerns regarding overfitting and poor generalizability.…”

Commentary: Validation of Machine Learning-Based Automated Surgical Instrument Annotation Using Publicly Available Intraoperative Video

“…Determining the external validation of a model can affect whether it will have a worthwhile impact clinically in multiple contexts. 14,15 Future studies involving machine learning tools should be conducted in multiple populations and/or use diverse data sets to properly investigate models' true accuracy and allow appropriate optimization. As advanced image analysis models, prediction analytics, risk calculators, and other decision tools become increasingly commonplace, external validation will be essential for effective clinical integration.…”

Commentary: Machine Learning to Predict Successful Opioid Dose Reduction or Stabilization After Spinal Cord Stimulation

“…9 In addition, as machine learning is becoming more widespread in neurosurgery, external validation and dissemination of testing platforms for such models need to be further emphasized to allow for the development of more usable prediction models in clinical practice. 17 Nonetheless, such studies are a step forward in the direction of better understanding chronic pain treatments and approaches to reduce opioid abuse in a time when it is of utmost importance.…”