Clinician Agreement on the Classification of Thyroid Nodules Ultrasound Features: A Survey of 2 Endocrine Societies

Abstract

Context Thyroid nodule risk stratification allows clinicians to standardize the evaluation of thyroid cancer risk according to ultrasound features. Objective To evaluate inter-rater agreement among clinicians assessing thyroid nodules ultrasound features and thyroid cancer risk categories. Design, setting, and participants We surveyed Endocrine Society and Latin American Thyroid Society members to assess their interpretation of composition, echogenicity, shape, margins, and presence of echogenic foci of ten thyroid nodule cases. The risk category for thyroid cancer was calculated following the ACR-TIRADS framework from individual responses. Main outcomes and measures We used descriptive statistics and Gwet's agreement coefficient (AC1) to assess the primary outcome of inter-rater agreement for ACR-TIRADS risk category. As secondary outomes the inter-rater agreement for individual features and subgroup analysis of inter-rater agreement for ACR-TIRADS category was performed (ultrasound reporting system, type of practice, and number of monthly appraisals). Results A total of 144 participants were included, mostly endocrinologists (82%). There was moderate level of agreement for the absence of echogenic foci (AC1 0.53, 95% confidence interval (CI) 0.24-0.81) and composition (AC1 0.54, 95%CI 0.36-0.71). The agreement for margins (AC1 0.24, 95% CI 0.15-0.33), echogenicity (AC1 0.34, 95% CI 0.22-0.46), and shape assessment (AC1 0.42, 95% CI 0.13-0.70) was lower. The overall agreement for ACR-TIRADS assessment was AC1 0.29, 95% CI 0.13-0.45. The AC1 of ACR-TIRADS among subgroups was similar. Conclusions This study found high variation of judgments about ACR-TIRADS risk category and individual features, which poses a potential challenge for the widescale implementation of thyroid nodule risk stratification.

Implementation strategies to support ultrasound thyroid nodule risk stratification: A systematic review

“…with variable training and systems in place to conduct thyroid nodule risk stratification, we found high variability of ACR TI-RADS categorisation (0.29, 95% CI: 0.13-0.45). 23 In all, this body of evidence supports the need for implementation strategies that can support not only adoption of thyroid nodule risk stratification as evidenced by the ultrasound report, but that also evaluate the accuracy and reproducibility of these assessments. Yet, only two of the studies included in this review described audit and feedback and quality assurance as implementation strategies.…”

“…25 Similarly, previous assessments of thyroid nodule risk stratification conducted by highly trained evaluators have shown moderate reproducibility with lower levels of agreement when a more diverse group of evaluators has been assessed. 23,24 Specifically, a meta-analysis of 7…”

“…19,21,22 Moreover, although clinical evidence from highly trained and experienced evaluators support acceptable measures of concordance on thyroid nodule risks assessment, measurements of reproducibility including more diverse groups of evaluators identify low concordance as an important concern in routine practice. [23][24][25] To facilitate adoption of high quality thyroid nodule risk stratification, it is important to identify implementation strategies that can support this process. In this systematic review, we aimed to summarise strategies used to promote high quality thyroid nodule risk stratification and assess the effects of these strategies on implementation success and service outcomes.…”

“…Some potential barriers to risk stratification use included lack of a universal lexicon, the need for additional training to support efficacy in thyroid ultrasound assessment, and time constraints 19,21,22 . Moreover, although clinical evidence from highly trained and experienced evaluators support acceptable measures of concordance on thyroid nodule risks assessment, measurements of reproducibility including more diverse groups of evaluators identify low concordance as an important concern in routine practice 23–25 …”

Inter‐Rater Reliability of Thyroid Ultrasound Risk Criteria: A Systematic Review and Meta‐Analysis

“…Of the 951 articles identified via the database search, 35 met eligibility criteria (Figure 1). 18–52 We found that the inter‐rater reliability amongst independent reviewers for the abstract screening and full‐text review stages was 0.704 (95% CI: 0.628, 0.781) and 0.766 (95% CI: 0.630, 0.903), respectively.…”