Diagnostic accuracy and interobserver variability of CO-RADS in patients with suspected coronavirus disease-2019: a multireader validation study

Abstract

Objective To conduct a multireader validation study to evaluate the interobserver variability and the diagnostic accuracy for the lung involvement by COVID-19 of COVID-19 Reporting and Data System (CO-RADS) score. Methods This retrospective study included consecutive symptomatic patients who underwent chest CT and reverse transcriptase-polymerase chain reaction (RT-PCR) from March 2020 to May 2020 for suspected COVID-19. Twelve readers with different levels of expertise independently scored each CT using the CO-RADS scheme for detecting pulmonary involvement by COVID-19. Receiver operating characteristic (ROC) curves were computed to investigate diagnostic yield. Fleiss’ kappa statistics was used to evaluate interreader agreement. Results A total of 572 patients (mean age, 63 ± 20 [standard deviation]; 329 men; 142 patients with COVID-19 and 430 patients without COVID-19) were evaluated. There was a moderate agreement for CO-RADS rating among all readers (Fleiss’ K = 0.43 [95% CI 0.42–0.44]) with a substantial agreement for CO-RADS 1 category (Fleiss’ K = 0.61 [95% CI 0.60–0.62]) and moderate agreement for CO-RADS 5 category (Fleiss’ K = 0.60 [95% CI 0.58–0.61]). ROC analysis showed the CO-RADS score ≥ 4 as the optimal threshold, with a cumulative area under the curve of 0.72 (95% CI 66–78%), sensitivity 61% (95% CI 52–69%), and specificity 81% (95% CI 77–84%). Conclusion CO-RADS showed high diagnostic accuracy and moderate interrater agreement across readers with different levels of expertise. Specificity is higher than previously thought and that could lead to reconsider the role of CT in this clinical setting. Key Points • COVID-19 Reporting and Data System (CO-RADS) demonstrated a good diagnostic accuracy for lung involvement by COVID-19 with an average AUC of 0.72 (95% CI 67 – 75%). • When a threshold of ≥ 4 was used, sensitivity and specificity were 61% (95% CI 52 – 69%) and 81% (95% CI 76 – 84%), respectively. • There was an overall moderate agreement for CO-RADS rating across readers with different levels of expertise (Fleiss’ K = 0.43 [95% CI 0.42 – 0.44]). Electronic supplementary material The online version of this article (10.1007/s00330-020-07273-y) contains supplementary material, which is available to authorized users.

Comparison of the CO-RADS and the RSNA chest CT classification system concerning sensitivity and reliability for the diagnosis of COVID-19 pneumonia

“…However, if there were real lobar pneumonias or "tree in bud" patterns, the agreement would have been higher. Our results are in line with those of several previous studies [ 15 , 23 , 28 , 31 , 32 ]. Prokop et al [ 15 ] conducted the first study that investigated the consistency of CO-RADS and reported a reasonable level of moderate intra-reviewer agreement ( κ = 0.47), with the highest agreement noted for CO-RADS 1 ( κ = 0.58) and 5 ( κ = 0.68).…”

“…Meanwhile, the sensitivity significantly increased for both systems when excluding the asymptomatic/mild patients and considering only moderate (sensitivity = 94.7%) and severe/critically ill patients (sensitivity = 97.8%); this is not surprising, taking into account that the sensitivity depends on CT imaging features, which have been considerably proven in several recent studies [ 6 , 18 – 22 ], while CT has been confirmed to be a reliable imaging approach for the evaluation of COVID-19. Our data are congruent with the results mentioned in previous research [ 15 , 23 – 28 ], which suggested that the CO-RADS and the RSNA chest CT classification system performed very well in predicting COVID-19 in patients with moderate to severe symptoms. Notably, a recent meta-analysis published by Kwee et al [ 29 ] concluded that COVID-19 infection frequency was higher in patients categorized with higher CO-RADS and RSNA classification categories.…”

“…Prokop et al [ 15 ] conducted the first study that investigated the consistency of CO-RADS and reported a reasonable level of moderate intra-reviewer agreement ( κ = 0.47), with the highest agreement noted for CO-RADS 1 ( κ = 0.58) and 5 ( κ = 0.68). A recent study published by Bellini et al [ 23 ] indicated a moderate level of overall agreement was obtained for CO-RADS ( κ = 0.43), with less agreement achieved for the intermediate (grades 2–4) CO-RADS categories than for CO-RADS 1 and 5. Separately, in a study conducted by Ciccarese et al [ 28 ], two readers evaluated 460 patients according to the RSNA chest CT classification system and achieved a good level of inter-reviewer agreement for the typical and negative categories and a fair level of inter-reviewer agreement for the indeterminate and atypical categories ( κ = 0.5).…”

Machine learning automatically detects COVID-19 using chest CTs in a large multicenter cohort

“…For reference, we have also included the performance of the manually rated CO-RADS scoring system by a radiologist on the same test set, which had sensitivity/specificity of 74%/71%. The reported performance of CO-RADS in identifying COVID-19 (positive RT-PCR) is variable in literature with Bellini et al reporting sensitivity/specificity of 71.0%/81.0% [18], Lieveld et al reporting sensitivity/specificity of 89.4%/87.2% [19], and Prokop et al reporting sensitivity/specificity of 90%/82.8% [16].…”

COVID-19 Imaging: What We Know Now and What Remains Unknown

“…A retrospective study of chest CT scans of 572 symptomatic patients (142 with RT-PCR confirmed COVID-I n p r e s s with a substantial agreement for CO-RADS 1 category (Fleiss' k = 0.61) and moderate agreement for CO-RADS 5 category (Fleiss' k = 0.60). CO-RADS score ≥ 4 was identified by ROC analysis as the optimal threshold with a cumulative area under the curve of 0.72, sensitivity 61%, and specificity 81% (42).…”