Analyzing the Quality and Challenges of Uncertainty Estimations for Brain Tumor Segmentation

Abstract

Automatic segmentation of brain tumors has the potential to enable volumetric measures and high-throughput analysis in the clinical setting. Reaching this potential seems almost achieved, considering the steady increase in segmentation accuracy. However, despite segmentation accuracy, the current methods still do not meet the robustness levels required for patient-centered clinical use. In this regard, uncertainty estimates are a promising direction to improve the robustness of automated segmentation systems. Different uncertainty estimation methods have been proposed, but little is known about their usefulness and limitations for brain tumor segmentation. In this study, we present an analysis of the most commonly used uncertainty estimation methods in regards to benefits and challenges for brain tumor segmentation. We evaluated their quality in terms of calibration, segmentation error localization, and segmentation failure detection. Our results show that the uncertainty methods are typically well-calibrated when evaluated at the dataset level. Evaluated at the subject level, we found notable miscalibrations and limited segmentation error localization (e.g., for correcting segmentations), which hinder the direct use of the voxel-wise uncertainties. Nevertheless, voxel-wise uncertainty showed value to detect failed segmentations when uncertainty estimates are aggregated at the subject level. Therefore, we suggest a careful usage of voxel-wise uncertainty measures and highlight the importance of developing solutions that address the subject-level requirements on calibration and segmentation error localization.

Post Training Uncertainty Calibration Of Deep Networks For Medical Image Segmentation

“…Surprisingly, models trained with soft Dice are better calibrated than those trained with cross-entropy for BraTS 2018. Consistent with previous results [11], we find that despite an improvement in average calibration performance, several subjects remain under-or overconfident. Using segmentation maps as feedback for uncertainty should therefore be done with caution.…”

“…Because of the bin-size weighting in the ECE metric, the often highly-confident and accurate background pixels have a large effect. We therefore only consider those voxels belonging to the brain for the ECE calculation, similar to [11].…”

Clinical target volume segmentation for stomach cancer by stochastic width deep neural network

“…This demonstrates that regularizing the shortcut connections in the segmentation network can achieve better performance. To evaluate whether the models above are certain about the segmentation results, we calculate the uncertainties 24 of all test cases and results are shown in Fig. 5, which shows the recall‐precision curve of the pixel‐level segmentation results.…”

Brain Tumor Segmentation via Multi-Modalities Interactive Feature Learning

“…where the d H ( x, y ) denotes the distance between pixels x ∈ P and y ∈ S . We follow the work ( 40 ) to use Euclidean distance to calculate the pixel-wise distance. The Hausdorff distance represents the longest distance from P (respectively S ) to its closest point in S (respectively P ).…”