A neural ordinary differential equation model for visualizing deep neural network behaviors in multi‐parametric MRI‐based glioma segmentation

Abstract

PurposeTo develop a neural ordinary differential equation (ODE) model for visualizing deep neural network behavior during multi‐parametric MRI‐based glioma segmentation as a method to enhance deep learning explainability.MethodsBy hypothesizing that deep feature extraction can be modeled as a spatiotemporally continuous process, we implemented a novel deep learning model, Neural ODE, in which deep feature extraction was governed by an ODE parameterized by a neural network. The dynamics of (1) MR images after interactions with the deep neural network and (2) segmentation formation can thus be visualized after solving the ODE. An accumulative contribution curve (ACC) was designed to quantitatively evaluate each MR image's utilization by the deep neural network toward the final segmentation results.The proposed Neural ODE model was demonstrated using 369 glioma patients with a 4‐modality multi‐parametric MRI protocol: T1, contrast‐enhanced T1 (T1‐Ce), T2, and FLAIR. Three Neural ODE models were trained to segment enhancing tumor (ET), tumor core (TC), and whole tumor (WT), respectively. The key MRI modalities with significant utilization by deep neural networks were identified based on ACC analysis. Segmentation results by deep neural networks using only the key MRI modalities were compared to those using all four MRI modalities in terms of Dice coefficient, accuracy, sensitivity, and specificity.ResultsAll Neural ODE models successfully illustrated image dynamics as expected. ACC analysis identified T1‐Ce as the only key modality in ET and TC segmentations, while both FLAIR and T2 were key modalities in WT segmentation. Compared to the U‐Net results using all four MRI modalities, the Dice coefficient of ET (0.784→0.775), TC (0.760→0.758), and WT (0.841→0.837) using the key modalities only had minimal differences without significance. Accuracy, sensitivity, and specificity results demonstrated the same patterns.ConclusionThe Neural ODE model offers a new tool for optimizing the deep learning model inputs with enhanced explainability. The presented methodology can be generalized to other medical image‐related deep‐learning applications.

Radiomics on spatial‐temporal manifolds via Fokker–Planck dynamics

“…Integration of radiomic analysis and other analytical tools that are mechanistically informed may increase both generalization and interpretation. [46][47][48][49][50] For example, radiomics-boosted deep learning models have been developed for diverse applications, such as COVID-19 pneumonia detection via chest radiographs, 51 post-resection survival prediction of patients with glioblastoma 50 and identification of radionecrosis following stereotactic radiosurgery (SRS) for brain metastases. 48 In each case, integration of radiomics and deep learning approaches serves to improve interpretability of deep learning models otherwise described as "black boxes".…”

“…This work seeked to integrate a traditional radiomics approach with techniques from applied mathematics and statistical mechanics to generate a new formalism. Integration of radiomic analysis and other analytical tools that are mechanistically informed may increase both generalization and interpretation 46–50 . For example, radiomics‐boosted deep learning models have been developed for diverse applications, such as COVID‐19 pneumonia detection via chest radiographs, 51 post‐resection survival prediction of patients with glioblastoma 50 and identification of radionecrosis following stereotactic radiosurgery (SRS) for brain metastases 48 .…”

On the Applications of Neural Ordinary Differential Equations in Medical Image Analysis

“…Evaluation results on the out-of-sample MRI dataset differ from the BraTS showing that the NODE helps improve the generalization capability of the segmentation network. Unlike the Sadique et al [35] that stacks the multi-parametric MRI along the channel dimension as input, Yang et al [38] utilizes a shared U-Net with NODE to process different modalities independently. The final tumor segmentation result is obtained through a weighted summation of the four branches corresponding to the four MRI modalities.…”

Quantifying U‐Net uncertainty in multi‐parametric MRI‐based glioma segmentation by spherical image projection

“…2,4,5 When translating research and development into real-world clinical applications, the robustness of deep neural network (DNN) predictions must be studied before incorporating it into patient care. [6][7][8] Classic neural networks are limited by their inability to deliver reliable uncertainty estimation and suffer from over-or under-confidence. 7 Specifically, in medical image segmentation, current DNNs learn from training cases (i.e., paired image data and segmentation ground truths derived from manual delineation) and make segmentation predictions using test cases.…”

“…Driven by recent developments in algorithms and increased computational power, deep learning has become the major vehicle for improved medical image segmentation 2,4,5 . When translating research and development into real‐world clinical applications, the robustness of deep neural network (DNN) predictions must be studied before incorporating it into patient care 6–8 . Classic neural networks are limited by their inability to deliver reliable uncertainty estimation and suffer from over‐ or under‐confidence 7 .…”