Comparison of Prostate MRI Lesion Segmentation Agreement Between Multiple Radiologists and a Fully Automatic Deep Learning System

Abstract

Purpose A recently developed deep learning model (U-Net) approximated the clinical performance of radiologists in the prediction of clinically significant prostate cancer (sPC) from prostate MRI. Here, we compare the agreement between lesion segmentations by U-Net with manual lesion segmentations performed by different radiologists. Materials and Methods 165 patients with suspicion for sPC underwent targeted and systematic fusion biopsy following 3 Tesla multiparametric MRI (mpMRI). Five sets of segmentations were generated retrospectively: segmentations of clinical lesions, independent segmentations by three radiologists, and fully automated bi-parametric U-Net segmentations. Per-lesion agreement was calculated for each rater by averaging Dice coefficients with all overlapping lesions from other raters. Agreement was compared using descriptive statistics and linear mixed models. Results The mean Dice coefficient for manual segmentations showed only moderate agreement at 0.48–0.52, reflecting the difficult visual task of determining the outline of otherwise jointly detected lesions. U-net segmentations were significantly smaller than manual segmentations (p < 0.0001) and exhibited a lower mean Dice coefficient of 0.22, which was significantly lower compared to manual segmentations (all p < 0.0001). These differences remained after correction for lesion size and were unaffected between sPC and non-sPC lesions and between peripheral and transition zone lesions. Conclusion Knowledge of the order of agreement of manual segmentations of different radiologists is important to set the expectation value for artificial intelligence (AI) systems in the task of prostate MRI lesion segmentation. Perfect agreement (Dice coefficient of one) should not be expected for AI. Lower Dice coefficients of U-Net compared to manual segmentations are only partially explained by smaller segmentation sizes and may result from a focus on the lesion core and a small relative lesion center shift. Although it is primarily important that AI detects sPC correctly, the Dice coefficient for overlapping lesions from multiple raters can be used as a secondary measure for segmentation quality in future studies. Key Points: Citation Format

Deep learning for emergency ascites diagnosis using ultrasonography images

“…Although correctly outlining lesion borders is valuable for treatment planning, determining the presence or absence of abdominal free fluid is the first step in our proposed methodology. The Dice coefficients achieved in this study reflect the difficulty of delineating ascites areas 58 . The low sensitivity can also be attributed to issues with detecting lesions in abdominal parenchymal organs.…”

“…Receiver operating characteristic (ROC) curve, sensitivity, specificity, accuracy, area under the curve (AUC), and average Dice coefficient were included as evaluation metrics. The Dice coefficient is a measure of contour consistency for segmented lesions occupying the same position and can be calculated as: $$\begin{equation*} \def\eqcellsep{&}\begin{array}{*{20}{c}} {Dice\; = \frac{{2\left| {X \cap Y} \right|}}{{\left| X \right| + \left| Y \right|}}\;\# \left( 1 \right)} \end{array} \end{equation*}$$where |X| and |Y | indicate the number of voxels in segmentations X and Y , and X ∩ Y defines the set of voxels that overlap between segmentations X and Y 57,58 . The ascites images after annotation are regarded as a reference and compared with the results of U‐net.…”

Role of Deep Learning in Prostate Cancer Management: Past, Present and Future Based on a Comprehensive Literature Review

“…The authors concluded that using DL models can allow for fast and accurate segmentation of MRI images from different scanners [22]. Schelb et al (2020) produced a comparison of prostate MRI lesion segmentation between a DL model and multiple radiologists. The study was performed using MRI images collected from 165 patients suspected to have prostate cancer.…”

“…The authors concluded that smaller segmentation sizes could explain the lower Dice coefficients of the U-Net model. They also discuss how the overlapping lesions between multiple rates can be used as a secondary measure for segmentation quality in future studies [23]. Soerensen et al (2021) performed a study to determine if DL improves the speed and accuracy of prostate gland segmentation from MRI images.…”

The impact of using voxel-level segmentation metrics on evaluating multifocal prostate cancer localisation

“…Recently, deep learning approaches have been proposed for diagnosing patients with suspicion of clinically significant prostate cancer and segment prostate lesions directly from mpMR images [5][8] [19][3] [23]. In addition to evaluating patient-level cancer detection -an image classification problem, results from image segmentation using UNet and its variant [17] have also been reported, as the mpMR-detected lesions may need histo-pathological examination or treatment, for example, through targeted biopsy [10] [13] and focal ablation [2] [15].…”