A Deep-Learning Diagnostic Support System for the Detection of COVID-19 Using Chest Radiographs

Abstract

Objectives: The aim of this study was to compare a diagnosis support system to detect COVID-19 pneumonia on chest radiographs (CXRs) against radiologists of various levels of expertise in chest imaging. Materials and Methods: Five publicly available databases comprising normal CXR, confirmed COVID-19 pneumonia cases, and other pneumonias were used. After the harmonization of the data, the training set included 7966 normal cases, 5451 with other pneumonia, and 258 CXRs with COVID-19 pneumonia, whereas in the testing data set, each category was represented by 100 cases. Eleven blinded radiologists with various levels of expertise independently read the testing data set. The data were analyzed separately with the newly proposed artificial intelligencebased system and by consultant radiologists and residents, with respect to positive predictive value (PPV), sensitivity, and F-score (harmonic mean for PPVand sensitivity). The χ 2 test was used to compare the sensitivity, specificity, accuracy, PPV, and F-scores of the readers and the system. Results: The proposed system achieved higher overall diagnostic accuracy (94.3%) than the radiologists (61.4% ± 5.3%). The radiologists reached average sensitivities for normal CXR, other type of pneumonia, and COVID-19 pneumonia of 85.0% ± 12.8%, 60.1% ± 12.2%, and 53.2% ± 11.2%, respectively, which were significantly lower than the results achieved by the algorithm (98.0%, 88.0%, and 97.0%; P < 0.00032). The mean PPVs for all 11 radiologists for the 3 categories were 82.4%, 59.0%, and 59.0% for the healthy, other pneumonia, and COVID-19 pneumonia, respectively, resulting in an F-score of 65.5% ± 12.4%, which was significantly lower than the F-score of the algorithm (94.3% ± 2.0%, P < 0.00001). When other pneumonia and COVID-19 pneumonia cases were pooled, the proposed system reached an accuracy of 95.7% for any pathology and the radiologists, 88.8%. The overall accuracy of consultants did not vary significantly compared with residents (65.0% ± 5.8% vs 67.4% ± 4.2%); however, consultants detected significantly more COVID-19 pneumonia cases (P = 0.008) and less healthy cases (P < 0.00001). Conclusions: The system showed robust accuracy for COVID-19 pneumonia detection on CXR and surpassed radiologists at various training levels.

A comprehensive review of imaging findings in COVID-19 - status in early 2021

“…The system performance was comparable with that of six independent readers. In a more recent study [299], the performance of a newly introduced DL-based system trained on publicly available datasets was compared against 11 radiologists for the three-way discriminatory diagnosis of COVID-19 pneumonia vs. other pneumonias or normal. The system detected COVID-19 very accurately and outperformed radiologists at various training levels, while it was able to separate COVID-19 pneumonia from other types of pneumonia more accurately than were the human readers.…”

Applications and challenges of AI-based algorithms in the COVID-19 pandemic

“…However, most of the published studies used relatively small datasets (<1000 CXR images of COVID-19 cases). [40][41][42][43][44] Transfer learning is an ML approach that can help investigators overcome limited data sizes. A CNN is pretrained with results of a previous training round from a different domain.…”

Chest Imaging of Patients with Sarcoidosis and SARS-CoV-2 Infection. Current Evidence and Clinical Perspectives

“…Deep learning techniques are giving promising results in assessing the radiological features of COVID-19 [61][62][63][64]. In particular, a recent study found that a proposed algorithm reached a significantly higher overall diagnostic accuracy than that obtained by a simple radiologist observation both in COVID-19 pneumonia than in pneumonia from other causes [65].…”