Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy

Abstract

Background and purpose: In radiotherapy, automatic organ-at-risk segmentation algorithms allow faster delineation times, but clinically relevant contour evaluation remains challenging. Commonly used measures to assess automatic contours, such as volumetric Dice Similarity Coefficient (DSC) or Hausdorff distance, have shown to be good measures for geometric similarity, but do not always correlate with clinical applicability of the contours, or time needed to adjust them. This study aimed to evaluate the correlation of new and commonly used evaluation measures with time-saving during contouring. Materials and methods: Twenty lung cancer patients were used to compare user-adjustments after atlas-based and deep-learning contouring with manual contouring. The absolute time needed (s) of adjusting the auto-contour compared to manual contouring was recorded, from this relative time-saving (%) was calculated. New evaluation measures (surface DSC and added path length, APL) and conventional evaluation measures (volumetric DSC and Hausdorff distance) were correlated with time-recordings and time-savings, quantified with the Pearson correlation coefficient, R. Results: The highest correlation (R = 0.87) was found between APL and absolute adaption time. Lower correlations were found for APL with relative time-saving (R = −0.38), for surface DSC with absolute adaption time (R = −0.69) and relative time-saving (R = 0.57). Volumetric DSC and Hausdorff distance also showed lower correlation coefficients for absolute adaptation time (R = −0.32 and 0.64, respectively) and relative timesaving (R = 0.44 and −0.64, respectively). Conclusion: Surface DSC and APL are better indicators for contour adaptation time and time-saving when using auto-segmentation and provide more clinically relevant and better quantitative measures for automaticallygenerated contour quality, compared to commonly-used geometry-based measures.

Assessment of manual adjustment performed in clinical practice following deep learning contouring for head and neck organs at risk in radiotherapy

“…Although it was expected that DLC would stop contouring at this point, it was also expected that manual adjustment would occur here depending on the PTV extent. DLC might be improved by contouring the whole spinal cord, as it is generally easier to delete than add [22] . However, such analysis would still demonstrate adjustment is required here as the guidelines accept some variation in where to stop.…”

“…As other studies have shown, variability in contouring does not result in significant dosimetric differences for the majority of organs [4] , [24] . A threshold for meaningful contour adjustments could possibly be set [22] , and RTTs trained to accept contours in cases not exceeding this threshold. Practical tools to guide the user in this decision-making process will enable further automation of contouring practice.…”

External validation of deep learning-based contouring of head and neck organs at risk

“…It would be good to benchmark the measures, because while DSC is reasonably straightforward, HD results can be influenced substantially by the calculation algorithm. It would be good to include surface DSC [24] and/or Added Path Length [25] , as these measures are a surrogate for the potential time saving of DLC in comparison to manual delineation. Valentini et al [2] recommended to evaluate time saving, together with an assessment of dosimetric impact.…”

Automatic evaluation of contours in radiotherapy planning utilising conformity indices and machine learning

“…Whilst some CIs used here are mathematically correlated, using them together does not impact upon a model’s ability to make predictions of new observations on unseen data [25] and helps increase model complexity to reduce the likelihood of underfitting. Alternatively, the use of other types of mathematical metrics, such as the surface DSC [26] and added path length [27] , could be investigated to determine their suitability for this purpose.…”