Background and Purpose To demonstrate and examine the ability of a newly developed software tool to estimate and analyze consensus contours from manually created contours by expert radiation oncologists. Material and Methods Several statistical methods and a graphical user interface were developed. For evaluation purposes, we used three breast cancer CT scans from the RTOG Breast Cancer Atlas Project. Specific structures were contoured before and after the experts’ consensus panel meeting. Differences in the contours were evaluated qualitatively and quantitatively by the consensus software tool. Estimates of consensus contours were analyzed for the different structures and Dice similarity and Dice-Jaccard coefficients were used for comparative evaluation. Results Based on kappa statistics, highest levels of agreement were seen in the left breast, lumpectomy, and heart. Significant improvements between pre- and post-consensus contours were seen in delineation of the chestwall and breasts while significant variations were noticed in the supraclavicular and internal mammary nodes. Dice calculations for all pre-consensus STAPLE estimations and final consensus panel structures reached 0.80 or greater for the heart, left/right breast, case-A lumpectomy, and chestwall. Conclusions Using the consensus software tool incorporating STAPLE estimates provided the ability to create contours similar to the ones generated by expert physicians.

Purpose: To analyze and validate the ability of a newly developed software tool to estimate and analyze consensus contours from manually delineated structures by panel of expert radiation oncologists for building reference atlases for radiotherapy and dosimetry practitioners. Method and Materials: Several statistical methods and a graphical user interface were developed as an integral part of an in‐house research treatment planning system. For evaluation purposes, we used three breast cancer CT scans from the RTOG Breast Cancer Atlas Project labeled A, B, and C. All participating physicians contoured specific structures before and after the consensus panel met without the software aid. Differences in the contours were analyzed qualitatively and quantitatively using metrics based on apparent agreement, kappa statistics, and STAPLE estimates by the developed consensus software tool. Estimates of the consensus contours were analyzed for the different structures before consensus and compared with the experts' manual consensus contour. Dice similarity coefficient (DSC) was used for comparative evaluation between the software and the experts' results. Results: Dice similarity calculations for all pre‐consensus STAPLE estimates and the final consensus panel structures reached 0.80 or greater for the heart, left breast, right breast, case A lumpectomy, case B lumpectomy, and the chest wall. The heart, case A lumpectomy, left breast, and right breast all exhibited a DSC greater than 0.90. The highest Dice coefficients were seen in the comparison of the STAPLE estimated left breast contours, which almost reached 0.96 at 100% confidence level. DSCs were in the range of .70 to .79 for the supraclavicular and internal mammary nodes. Conclusion: Using the consensus software tool incorporating several statistical methods allowed excellent ability to create contours similar to the ones generated by expert physicians and provided metrics for the confidence levels of agreement.