Interobserver Variability of Ultrasound Features Based on American College of Radiology Thyroid Imaging Reporting and Data System Lexicon in American College of Radiology Thyroid Imaging Reporting and Data System System

Abstract

The aim of this study is to evaluate the variability of selecting the ultrasound features used in American College of Radiology Thyroid Imaging Reporting and Data System (ACR-TIRADS) and in assigning the ACR-TIRADS level in a single center among radiologists and radiology residents. The study cohort consisted of 108 thyroid nodules in 102 patients who had definite cytology results after thyroid fine needle aspiration biopsy (Bethesda category II, VI) or surgery. Seven observers including 3 radiologists and 4 radiology residents evaluated the nodules according to 5 ultrasound feature categories. The evaluation process was performed after a joint meeting session, in which the "white papers" of the ACR-TIRADS committee were discussed regarding the thyroid ultrasound reporting lexicon, and final TIRADS system. Variability of ultrasound features and assigning ACR-TIRADS level was measured using Fleiss kappa statistics. Agreement for ultrasound features was "substantial" to "almost perfect" among all observers, with composition (κ = 0.86), macrocalcification (κ = 0.89) and peripheral calcification (κ = 0.92) at the highest level of agreement. The level of agreement for large comet tail artifacts and punctate echogenic foci was "moderate" in residents, whereas in radiologists, that level was "substantial." The agreement for assigning ACR-TIRADS level was moderate in resident as well as in radiologist subgroup. Agreement of thyroid ultrasound features was "substantial to almost perfect" among all observers. Although the level of agreement among resident group decreased to "moderate" level, ACR-TIRADS is a useful system in thyroid nodule management.

Considerable interobserver variation calls for unambiguous definitions of thyroid nodule ultrasound characteristics

“…Based on our findings, interobserver agreement was insufficient for evaluation of nodule margins and moderate for microcalcifications, a clear difference compared to previous studies which found better agreement (19,(31)(32)(33)(34)(35)(36)(37)(38)(39)(40)(41)(42)(43)(44)(45)(46)(47)(48)(49) using still images (Figure 2), while intraobserver variation was comparable (Figure 3). This is explained almost exclusively by the difference in observer-dependent interpretation of nodule characteristics; while microcalcification and taller-than-wide shape have clear definitions, this is less true for nodule margins.…”

“…Based on our findings, interobserver agreement was insufficient for the evaluation of nodule margins and moderate for microcalcifications, a clear difference compared to previous studies which found better agreement ( 19 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 ) using still images ( Fig. 2 ), while intraobserver variation was comparable ( Fig.…”

“…The diagnostic sensitivity and specificity of suspicious ultrasound characteristics for predicting thyroid cancer. Comparison with the previously published data(25,(37)(38)(39)(40)(41)(42)(43)(44)(45)(46)(47)(48)(49).SupplementaryTable. Questions answered by the investigators for each nodule 1.…”

Inter‐Rater Reliability of Thyroid Ultrasound Risk Criteria: A Systematic Review and Meta‐Analysis

“…Of the 951 articles identified via the database search, 35 met eligibility criteria (Figure 1). 18–52 We found that the inter‐rater reliability amongst independent reviewers for the abstract screening and full‐text review stages was 0.704 (95% CI: 0.628, 0.781) and 0.766 (95% CI: 0.630, 0.903), respectively.…”