Interobserver variability in organ at risk delineation in head and neck cancer

Abstract

Background In radiotherapy inaccuracy in organ at risk (OAR) delineation can impact treatment plan optimisation and treatment plan evaluation. Brouwer et al. showed significant interobserver variability (IOV) in OAR delineation in head and neck cancer (HNC) and published international consensus guidelines (ICG) for OAR delineation in 2015. The aim of our study was to evaluate IOV in the presence of these guidelines. Methods HNC radiation oncologists (RO) from each Belgian radiotherapy centre were invited to complete a survey and submit contours for 5 HNC cases. Reference contours (OARref) were obtained by a clinically validated artificial intelligence-tool trained using ICG. Dice similarity coefficients (DSC), mean surface distance (MSD) and 95% Hausdorff distances (HD95) were used for comparison. Results Fourteen of twenty-two RO (64%) completed the survey and submitted delineations. Thirteen (93%) confirmed the use of delineation guidelines, of which six (43%) used the ICG. The OARs whose delineations agreed best with the OARref were mandible [median DSC 0.9, range (0.8–0.9); median MSD 1.1 mm, range (0.8–8.3), median HD95 3.4 mm, range (1.5–38.7)], brainstem [median DSC 0.9 (0.6–0.9); median MSD 1.5 mm (1.1–4.0), median HD95 4.0 mm (2.3–15.0)], submandibular glands [median DSC 0.8 (0.5–0.9); median MSD 1.2 mm (0.9–2.5), median HD95 3.1 mm (1.8–12.2)] and parotids [median DSC 0.9 (0.6–0.9); median MSD 1.9 mm (1.2–4.2), median HD95 5.1 mm (3.1–19.2)]. Oral cavity, cochleas, PCMs, supraglottic larynx and glottic area showed more variation. RO who used the consensus guidelines showed significantly less IOV (p = 0.008). Conclusions Although ICG for delineation of OARs in HNC exist, they are only implemented by about half of RO participating in this study, which partly explains the delineation variability. However, this study highlights that guidelines alone do not suffice to eliminate IOV and that more effort needs to be done to accomplish further treatment standardisation, for example with artificial intelligence.

Inter-observer variability of organ contouring for preclinical studies with cone beam Computed Tomography imaging

“…While arguably the number of observers (n = 2) is small compared to five or more that are commonly used in human contouring studies, for preclinical studies it is hard to find multiple contouring experts. As shown in previous studies, the IOVs are even larger depending on the annotators experience or the guidelines resulting in uncertainties when delineating normal tissue [8] , [15] . Human structures are also larger compared to the rodents and the findings in small animals with high spatial resolution during the CT acquisition need to be taken into account to compared to human studies, e.g., 10–50 times higher spatial CBCT resolution for rodents compared to humans.…”

“…Similarly, extensive work has been conducted on the development of advanced treatment-planning systems combined with high precision irradiation platforms mimicking human RT [4] , [5] , [6] . Even though there is extensive research in humans regarding the quantification of the interobserver variability (IOV) in organ segmentation [7] , [8] , similar research in small animals has only recently been started [9] .…”

Target Definition in MR-Guided Adaptive Radiotherapy for Head and Neck Cancer

“…A CT-based atlas is used for delineation of organs at risk (OARs) [ 39 ]. Although the atlas caused fairly good conformity in the delineation of parotid glands, submandibular glands and spinal cord, there still remains variability in CT-based delineation of the pharyngeal constrictor muscles and oral cavity [ 10 ]. In particular, the radiation dose to the pharyngeal constrictor muscles, oral cavity and salivary glands play a crucial role in late radiation-induced toxicity [ 8 , 40 , 41 ], so accurate delineation (on MRI) is essential [ 42 , 43 ].…”

“…Most crucial is the delineation of the tumor and OARs. Although delineation of the OARs is standardized and, in general, conformity index is acceptable in studies [ 9 , 10 ], delineation of the tumor, and less prominent delineation of the neck nodes may show more interobserver variation among radiation oncologists [ 11 ]. Improvement in validated imaging techniques is mandatory.…”

Pelvic U-Net: multi-label semantic segmentation of pelvic organs at risk for radiation therapy anal cancer patients using a deeply supervised shuffle attention convolutional neural network

“…This manual delineation operation can produce semantic segmentations, where each pixel, or voxel for 3D volumes, is classified as belonging to a set of predefined classes. Nevertheless, manual delineation is a time-consuming process and suffers from inter-observer variability [1][2][3][4].…”