Validation of Automatic Contour Propagation for 4D Treatment Planning Using Multiple Metrics

Abstract

The aim of this work is to provide insights into multiple metrics clinical validation of deformable image registration and contour propagation methods in 4D lung radiotherapy planning. The following indices were analyzed and compared: Volume Difference (VD), Dice Similarity Coefficient (DSC), Positive Predictive Value (PPV) and Surface Distances (SD). The analysis was performed on three patient datasets, using as reference a ground-truth volume generated by means of Simultaneous Truth And Performance Level Estimation (STAPLE) algorithm from the outlines of five experts. Significant discrepancies in the quality assessment provided by the different metrics in all the examined cases were found. Metrics sensitivity was more evident in presence of image artifacts and particularly for tubular anatomical structures, such as esophagus or spinal cord. Volume Differences did not account for position and DSC exhibited criticalities due to its intrinsic symmetry (i.e. over-and under-estimation of the reference contours cannot be discriminated) and dependency on the total volume of the structure. PPV analysis showed more robust performance, as each voxel concurs to the classification of the propagation, but was not able to detect inclusion of propagated and ground-truth volumes.Mesh distances could interpret the actual shape of the structures, but might report higher mismatches in case of large local differences in the contour surfaces. According to our study, the combination of VD and SD for the validation of contour propagation algorithms in 4D could provide the necessary failure detection accuracy.

Evaluation of Image Registration Accuracy for Tumor and Organs at Risk in the Thorax for Compliance With TG 132 Recommendations

“…28 , 29 The cord, heart, and lung tumor DSC values using segmentation or propagation have also been reported to range between 0.8 and 0.9 for clinically acceptable results. 23 , 24 Values for the quantitative accuracy metrics in Figure 4 of the present study were in agreement with these previously reported findings.…”

“…In an interobserver contouring investigation, DSC for thoracic structures was reported to range between 0.74 for esophagus and 0.96 for lungs, with the esophagus generating the largest interobserver variability 13 . Other studies report esophagus DSC ranging from 0.7 to 0.8 and HD of approximately 10 mm 23, 24, 25. Studies reporting lung contour propagation accuracy have found MDA, HD, and DSC values ranging from 1.5 to 2 mm, 20 to 25 mm, and 0.91 to 0.96, respectively 4, 23, 24, 26, 27.…”

“… 13 Other studies report esophagus DSC ranging from 0.7 to 0.8 and HD of approximately 10 mm. 23 , 24 , 25 Studies reporting lung contour propagation accuracy have found MDA, HD, and DSC values ranging from 1.5 to 2 mm, 20 to 25 mm, and 0.91 to 0.96, respectively. 4 , 23 , 24 , 26 , 27 For lung, a DSC >0.9 is expected in instances of acceptable agreement.…”

“… 23 , 24 , 25 Studies reporting lung contour propagation accuracy have found MDA, HD, and DSC values ranging from 1.5 to 2 mm, 20 to 25 mm, and 0.91 to 0.96, respectively. 4 , 23 , 24 , 26 , 27 For lung, a DSC >0.9 is expected in instances of acceptable agreement. 26 Automatic segmentation of vertebrae has been found to result in contours with DSC values between 0.8 and 0.95 compared with ground-truth contours, depending on the segmentation or propagation method used.…”

Evaluation of automatic contour propagation in T2‐weighted 4DMRI for normal‐tissue motion assessment using internal organ‐at‐risk volume (IRV)

“…However, the intra‐ and inter‐observer variation is common, leading to multiple ground truths. To minimize the uncertainty in the scientific ground truth, the simultaneous truth and performance level estimation ( STAPLE) algorithm was developed40 and applied to evaluate tumor delineation in radiotherapy planning of lung, liver, and pancreatic cancer using CT or 4DCT 41, 42, 43, 44. Based on the statistics of a group of clinical ground truths from multiple physicians, the most probable ground truth can be computed and used as the scientific ground truth.…”