Application of simultaneous uncertainty quantification for image segmentation with probabilistic deep learning: Performance benchmarking of oropharyngeal cancer target delineation as a use-case

Abstract

Background: Oropharyngeal cancer (OPC) is a widespread disease, with radiotherapy being a core treatment modality. Manual segmentation of the primary gross tumor volume (GTVp) is currently employed for OPC radiotherapy planning, but is subject to significant interobserver variability. Deep learning (DL) approaches have shown promise in automating GTVp segmentation, but comparative (auto)confidence metrics of these models predictions has not been well-explored. Quantifying instance-specific DL model uncertainty is crucial to improving clinician trust and facilitating broad clinical implementation. Therefore, in this study, probabilistic DL models for GTVp auto-segmentation were developed using large-scale PET/CT datasets, and various uncertainty auto-estimation methods were systematically investigated and benchmarked. Methods: We utilized the publicly available 2021 HECKTOR Challenge training dataset with 224 co-registered PET/CT scans of OPC patients with corresponding GTVp segmentations as a development set. A separate set of 67 co-registered PET/CT scans of OPC patients with corresponding GTVp segmentations was used for external validation. Two approximate Bayesian deep learning methods, the MC Dropout Ensemble and Deep Ensemble, both with five submodels, were evaluated for GTVp segmentation and uncertainty performance. The segmentation performance was evaluated using the volumetric Dice similarity coefficient (DSC), mean surface distance (MSD), and Hausdorff distance at 95% (95HD). The uncertainty was evaluated using four measures from literature: coefficient of variation (CV), structure expected entropy, structure predictive entropy, and structure mutual information, and additionally with our novel Dice-risk measure. The utility of uncertainty information was evaluated with the accuracy of uncertainty-based segmentation performance prediction using the Accuracy vs Uncertainty (AvU) metric, and by examining the linear correlation between uncertainty estimates and DSC. In addition, batch-based and instance-based referral processes were examined, where the patients with high uncertainty were rejected from the set. In the batch referral process, the area under the referral curve with DSC (R-DSC AUC) was used for evaluation, whereas in the instance referral process, the DSC at various uncertainty thresholds were examined. Results: Both models behaved similarly in terms of the segmentation performance and uncertainty estimation. Specifically, the MC Dropout Ensemble had 0.776 DSC, 1.703 mm MSD, and 5.385 mm 95HD. The Deep Ensemble had 0.767 DSC, 1.717 mm MSD, and 5.477 mm 95HD. The uncertainty measure with the highest DSC correlation was structure predictive entropy with correlation coefficients of 0.699 and 0.692 for the MC Dropout Ensemble and the Deep Ensemble, respectively. The highest AvU value was 0.866 for both models. The best performing uncertainty measure for both models was the CV which had R-DSC AUC of 0.783 and 0.782 for the MC Dropout Ensemble and Deep Ensemble, respectively. With referring patients based on uncertainty thresholds from 0.85 validation DSC for all uncertainty measures, on average the DSC improved from the full dataset by 4.7% and 5.0% while referring 21.8% and 22% patients for MC Dropout Ensemble and Deep Ensemble, respectively. Conclusion: We found that many of the investigated methods provide overall similar but distinct utility in terms of predicting segmentation quality and referral performance. These findings are a critical first-step towards more widespread implementation of uncertainty quantification in OPC GTVp segmentation.

Uncertainty-Aware Deep Learning for Segmentation of Primary Tumour and Pathologic Lymph Nodes in Oropharyngeal Cancer: Insights from a Multi-Centre Cohort

“…Secondly, this study employed only one metric to quantify uncertainty. The choice was made because this metric aligns well with our threshold-based approach and was shown to be the preferred choice for GTVp segmentation tasks by Sahlsten et al [5]. Nevertheless, a broader exploration of patient-specific characteristics is required to gain a comprehensive understanding of their effects on DL segmentation model results.…”

“…To quantify uncertainty, Sahlsten et al demonstrated that the coefficient of variation (CV) may be optimal as an uncertainty measure in OPC tumour segmentation [5]. This metric quantifies the variation of the volume across multiple predictions, and it showed to be negatively correlated with the DSC in GTVp segmentation.…”

Harnessing uncertainty in radiotherapy auto-segmentation quality assurance

“…Importantly, ensembling (e.g., through cross-validation schemes) is becoming increasingly common for many DL solutions [9] . We have previously benchmarked ensembling under a U-net framework for uncertainty estimation in oropharyngeal cancer auto-segmentation and have shown its efficacy [10] . Interestingly, Outeiral et al use cross-validation within their study for robustness analysis; merging their cross-validation outputs into an ensemble could have improved calibration when employing their HiS metric.…”

“…Finally, we would like to note that the proposed HiS metric, if used to measure uncertainty, may be unable to disentangle epistemic uncertainty (i.e., intrinsic model uncertainty) and aleatoric uncertainty (i.e., extrinsic statistical uncertainty) [12] . While the same can be said of general measures of entropy, there exist alternative entropy-related uncertainty metrics, like expected entropy and mutual information, that could distinguish the source of the uncertainty when combined with an approximate Bayesian approach [10] , [13] . Moreover, when the distribution of DL network parameters is assumed to be a delta distribution, e.g., in a conventional DL network, the epistemic uncertainty is implicitly assumed to be non-existent.…”

“…An increasing number of studies have begun to apply uncertainty estimation to the quality assurance of radiotherapy-related auto-segmentation [10] , [14] , [15] , [16] , [17] , [18] , [19] , [20] , [21] , [22] , [23] . The study by Outeiral et al serves as a cornerstone contribution to this crucial literature.…”