Incremental retraining, clinical implementation, and acceptance rate of deep learning auto‐segmentation for male pelvis in a multiuser environment

Abstract

Background Deep learning auto‐segmentation (DLAS) models have been adopted in the clinic; however, they suffer from performance deterioration owing to the clinical practice variability. Some commercial DLAS software provide an incremental retraining function that enables users to train a custom model using their institutional data to account for clinical practice variability. Purpose This study was performed to evaluate and implement the commercial DLAS software with the incremental retraining function for definitive treatment of patients with prostate cancer in a multi‐user environment. Methods CT‐based target organs and organs‐at‐risk (OAR) delineation of 215 prostate cancer patients were utilized. The performance of three commercial DLAS software built‐in models was validated with 20 patients. A retrained custom model was developed using 100 patients and evaluated on the remaining data (n = 115). Dice similarity coefficient (DSC), Hausdorff distance (HD), mean surface distance (MSD), and surface DSC (SDSC) were utilized for quantitative evaluation. A multi‐rater qualitative evaluation was blindly performed with a five‐level scale. Visual inspection was performed in consensus and non‐consensus unacceptable cases to identify the failure modes. Results Three commercial DLAS vendor built‐in models achieved sub‐optimal performance in 20 patients. The retrained custom model had a mean DSC of 0.82 for prostate, 0.48 for seminal vesicles (SV), and 0.92 for rectum, respectively. This represents a significant improvement over the built‐in model with DSC of 0.73, 0.37, and 0.81 for the corresponding structures. Compared to the acceptance rate of 96.5% and consensus unacceptable rate (i.e., both reviewers rated as unacceptable) of 3.5% achieved by manual contours, the custom model achieved a 91.3% acceptance rate and 8.7% consensus unacceptable rate. The failure modes of retrained custom model were attributed to the following: cystogram (n = 2), hip prosthesis (n = 2), low dose rate brachytherapy seeds (n = 2), air in endorectal balloon(n = 1), non‐iodinated spacer (n = 2), and giant bladder(n = 1). Conclusion The commercial DLAS software with the incremental retraining function was validated and clinically adopted for prostate patients in a multi‐user environment. AI‐based auto‐delineation of the prostate and OARs is shown to achieve improved physician acceptance, overall clinical utility, and accuracy.

Contour subregion error detection methodology using deep learning auto‐segmentation

“…It is highly possible that the contour labels from different ROs may vary due to their distinct training backgrounds, experiences, personal preferences, and other factors. 26,27 The inherent bias in human labels, on the other hand, can partially affect the selection of z values, which determine the pass criteria. When multiple ROs collaborate to define acceptability, some regions of the contour may demonstrate larger tolerances, with the acceptable deviation increasing to accommodate the varying perspectives of the ROs.…”

“…The impact of human subjectivity on the CSED system still remains uncertain even though a recent study revealed that bias in the labeling process can be mitigated by comparing the labels from multiple experts. 26 Future studies will be conducted by gathering different ROs' judgments on the same dataset and evaluating the effect of human bias in the proposed model.…”

“…While DLAS has showcased impressive performance in numerous studies, 21,[32][33][34][35] concerns regarding the generalizability and potential for generating clinically unacceptable contours have emerged. 26,29 Contour failures can occur when the DLAS is applied to an unseen dataset, where clinical practice deviates from the training dataset. 26,29,30 We noticed a significant degradation in DLAS performance in a previous study when it was directly applied to a local dataset using abdominal compression compression.…”

“…26,29 Contour failures can occur when the DLAS is applied to an unseen dataset, where clinical practice deviates from the training dataset. 26,29,30 We noticed a significant degradation in DLAS performance in a previous study when it was directly applied to a local dataset using abdominal compression compression. 30 Such a phenomenon emphasizes the need for automated QA tools to ensure DLAS quality.…”

“…Several studies have shown that DLAS may not produce accurate contours consistently in all cases. 26,29,30 The performance of DLAS may vary with clinical scenarios or clinical practices, resulting in sub-optimal contours. An inaccurate reference contour, such as the brainstem contour in Figure 6a, can introduce errors causing undesirable false positives.…”