Comparing a deep learning model’s diagnostic performance to that of radiologists to detect Covid -19 features on chest radiographs

Abstract

Background: Whether the sensitivity of Deep Learning (DL) models to screen chest radiographs (CXR) for CoVID-19 can approximate that of radiologists, so that they can be adopted and used if real-time review of CXRs by radiologists is not possible, has not been explored before. Objective: To evaluate the diagnostic performance of a doctor-trained DL model (Svita_DL8) to screen for COVID-19 on CXR, and to compare the performance of the DL model with that of expert radiologists. Materials and Methods: We used a pre-trained convolutional neural network to develop a publicly available online DL model to evaluate CXR examinations saved in .jpeg or .png format. The initial model was subsequently curated and trained by an internist and a radiologist using 1062 chest radiographs to classify a submitted CXR as either normal, COVID-19, or a non-COVID-19 abnormal. For validation, we collected a separate set of 430 CXR examinations from numerous publicly available datasets from 10 different countries, case presentations, and two hospital repositories. These examinations were assessed for COVID-19 by the DL model and by two independent radiologists. Diagnostic performance was compared between the model and the radiologists and the correlation coefficient calculated. Results: For detecting COVID-19 on CXR, our DL model demonstrated sensitivity of 91.5%, specificity of 55.3%, PPV 60.9%, NPV 77.9%, accuracy 70.1%, and AUC 0.73 (95% CI: 0.86, 0.95). There was a significant correlation (r = 0.617, P = 0.000) between the results of the DL model and the radiologists’ interpretations. The sensitivity of the radiologists is 96% and their overall diagnostic accuracy is 90% in this study. Conclusions: The DL model demonstrated high sensitivity for detecting COVID-19 on CXR. Clinical Impact: The doctor trained DL tool Svita_DL8 can be used in resource-constrained settings to quickly triage patients with suspected COVID-19 for further in-depth review and testing.

Artificial intelligence for imaging-based COVID-19 detection: Systematic review comparing added value of AI versus human readers

“…A total of 20 studies that met these criteria were assessed for the risk of bias with the QUADAS-2 tool. [5] , [10] , [11] , [12] , [13] , [14] , [15] , [16] , [17] , [18] , [19] , [20] , [21] , [22] , [23] , [24] , [25] , [26] , [27] , [28] Eight studies with a high risk of bias rating in at least two domains were excluded ( figure 1 , appendix, Table 3. ).…”

Radiology Implementation Considerations for Artificial Intelligence (AI) Applied to COVID-19, From the AJR Special Series on AI Applications

“…In one study evaluating the impact of an AI tool on radiologist performance, radiologists using an AI tool had superior sensitivity and PPV for COVID-19 compared to image interpretation without AI [16]. However, other studies have shown worse accuracy of some AI models compared to radiologists for the diagnosis of COVID-19 on chest radiographs [17,18]. AI applied to CT has also shown promise.…”

Discovery of a Generalization Gap of Convolutional Neural Networks on COVID-19 X-Rays Classification

“…Performance Result Testing Dataset Ozturk et al [7] 0.99 AUC train/test split from the same data source Abbas et al [8] 0.94 AUC train/test split from the same data source Farooq et al [9] 96.23% Accuracy train/test split from the same data source Lv et al [10] 85.62% Accuracy train/test split from the same data source Bassi and Attux [11] 97.80% Recall train/test split from the same data source Rahimzadeh and Attar [12] 99.60% Accuracy train/test split from the same data source Chowdhury et al [13] 98.30% Accuracy train/test split from the same data source Hemdan et al [14] 0.89 F1-score train/test split from the same data source Karim et al [15] 83.00% Recall train/test split from the same data source Krishnamoorthy et al [16] 90% Accuracy train/test split from the same data source Minaee et al [17] 90% Specificity train/test split from the same data source Afshar et al [18] 98.3% Accuracy train/test split from the same data source Khobahi et al [19] 93.5% Accuracy train/test split from the same data source Moura et al [20] 90.27% Accuracy train/test split from the same data source Kumar et al [21] 97.7% Accuracy train/test split from the same data source Castiglioni et al [22] 0.80 AUC train/test split from the same data source Rahaman et al [23] f 89.3% Accuracy train/test split from the same data source Hall et al [24] 0.95 AUC 10-fold cross validation with all sources mixed Apostolopoulos et al [25] 92.85% Accuracy 10-fold cross validation with all sources mixed Apostolopoulos et al [26] 99.18% Accuracy 10-fold cross validation with all sources mixed Mukherjee et al [27] 0.9908 AUC 10-fold cross validation with all sources mixed Das et al [28] 1.00 AUC 10-fold cross validation with all sources mixed Razzak et al [29] 98.75% Accuracy 10-fold cross validation with all sources mixed Basu et al [30] 95.30% Accuracy 5-fold cross validation with all sources mixed Li et al [31] 97.01% Accuracy 5-fold cross validation with all sources mixed Mukherjee et al [32] 0.9995 AUC 5-fold cross validation with all sources mixed Moutounet-Cartan et al [33] 93.9% Accuracy 5-fold cross validation with all s...…”