External Validation of SpineNet, an Open-Source Deep Learning Model for Grading Lumbar Disk Degeneration MRI Features, Using the Northern Finland Birth Cohort 1966

Abstract

Study Design. This is a retrospective observational study to externally validate a deep learning image classification model. Objective. Deep learning models such as SpineNet offer the possibility of automating the process of disk degeneration (DD) classification from magnetic resonance imaging (MRI). External validation is an essential step to their development. The aim of this study was to externally validate SpineNet predictions for DD using Pfirrmann classification and Modic changes (MCs) on data from the Northern Finland Birth Cohort 1966 (NFBC1966). Summary of Data. We validated SpineNet using data from 1331 NFBC1966 participants for whom both lumbar spine MRI data and consensus DD gradings were available. Materials and Methods. SpineNet returned Pfirrmann grade and MC presence from T2-weighted sagittal lumbar MRI sequences from NFBC1966, a data set geographically and temporally separated from its training data set. A range of agreement and reliability metrics were used to compare predictions with expert radiologists. Subsets of data that match SpineNet training data more closely were also tested. Results. Balanced accuracy for DD was 78% (77%–79%) and for MC 86% (85%–86%). Interrater reliability for Pfirrmann grading was Lin concordance correlation coefficient=0.86 (0.85–0.87) and Cohen κ=0.68 (0.67–0.69). In a low back pain subset, these reliability metrics remained largely unchanged. In total, 20.83% of disks were rated differently by SpineNet compared with the human raters, but only 0.85% of disks had a grade difference >1. Interrater reliability for MC detection was κ=0.74 (0.72–0.75). In the low back pain subset, this metric was almost unchanged at κ=0.76 (0.73–0.79). Conclusions. In this study, SpineNet has been benchmarked against expert human raters in the research setting. It has matched human reliability and demonstrates robust performance despite the multiple challenges facing model generalizability.

Semiautomatic Assessment of Facet Tropism From Lumbar Spine MRI Using Deep Learning

“…Although we observed a high inter-rater correlation in asymmetry ( r 2 =0.84) between radiologists (Fig. 6), future works may benefit from incorporating methods that account for raters’ 41,42 and center 43 variability. Second, the demographics of NFBC1966 are confined to the Finnish population and may not account for the variation of asymmetry in relation to the subject’s age, race, etc .…”

Comparing image normalization techniques in an end-to-end model for automated modic changes classification from MRI images

“…This is due to the use of T2-weighted images only that does not allow to discriminate MC-1 and MC-2. Subsequently, SpineNet was validated on an external validation set giving an accuracy of around 86% ( McSweeney et al, 2023 ). Similar to our work, another study aimed at classifying MCs into three subtypes namely MC-0, MC-1, and MC-2 ( Damopoulos et al, 2019 ).…”

Advancing spine care through AI and machine learning: overview and applications

“…Besides localizing all visible vertebral bodies in the image stack, SpineNet can evaluate several parameters for each vertebral level: the Pfirrmann grade of disc degeneration, presence of disc narrowing, presence and severity of central canal stenosis, endplate defect and marrow changes, foraminal stenosis, spondylolisthesis, and disc herniation. Two distinct recent studies performed by independent research groups described an external validation of SpineNet, demonstrating its robustness and reliability (64,65).…”