Diagnostic Performance of a Deep Learning Model Deployed at a National COVID-19 Screening Facility for Detection of Pneumonia on Frontal Chest Radiographs

Abstract

(1) Background: Chest radiographs are the mainstay of initial radiological investigation in this COVID-19 pandemic. A reliable and readily deployable artificial intelligence (AI) algorithm that detects pneumonia in COVID-19 suspects can be useful for screening or triage in a hospital setting. This study has a few objectives: first, to develop a model that accurately detects pneumonia in COVID-19 suspects; second, to assess its performance in a real-world clinical setting; and third, by integrating the model with the daily clinical workflow, to measure its impact on report turn-around time. (2) Methods: The model was developed from the NIH Chest-14 open-source dataset and fine-tuned using an internal dataset comprising more than 4000 CXRs acquired in our institution. Input from two senior radiologists provided the reference standard. The model was integrated into daily clinical workflow, prioritising abnormal CXRs for expedited reporting. Area under the receiver operating characteristic curve (AUC), F1 score, sensitivity, and specificity were calculated to characterise diagnostic performance. The average time taken by radiologists in reporting the CXRs was compared against the mean baseline time taken prior to implementation of the AI model. (3) Results: 9431 unique CXRs were included in the datasets, of which 1232 were ground truth-labelled positive for pneumonia. On the “live” dataset, the model achieved an AUC of 0.95 (95% confidence interval (CI): 0.92, 0.96) corresponding to a specificity of 97% (95% CI: 0.97, 0.98) and sensitivity of 79% (95% CI: 0.72, 0.84). No statistically significant degradation of diagnostic performance was encountered during clinical deployment, and report turn-around time was reduced by 22%. (4) Conclusion: In real-world clinical deployment, our model expedites reporting of pneumonia in COVID-19 suspects while preserving diagnostic performance without significant model drift.

Combating Covid-19 using machine learning and deep learning: Applications, challenges, and future perspectives

“… COVID-CXNet(Accuracy 87.88%) Nassif et al [ 59 ],(2022) Detection CXR,Audio 1159 sound samples 13,808 CXR Image LSTM, VGG16, VGG19, DENSNET201,RESNET50, INCEPTIONV3 INCEPTIONRESNETV2, XCEPTION LSTM (Accuracy of 98%)VGG16(Accuracy 89.64%)InceptionResNetV2(Accuracy 82.22%) Nayak et al [ 60 ],(2020) Detection CXR 406 ALEXNET,VGG16,GOOGLE NET,MOBILE NET-V2,SQUEEZENET,RESNET-34, RESNET-50,INCEPTION-V3 ResNet-34 (Accuracy 98.33%). Verma et al [ 61 ],(2022) Detection CT 63,849 RESNET50 V2, EFFICIENTNET B0 EfficientNet B0 (Sensitivity 99.69%) Sim et al [ 62 ],(2022) Detection CXR 5717 DENSENET121 DenseNet121(Sensitivity 95%) Srivastava and Ruchilekha [ 63 ],(2022) Detection CXR, CT 4271 DEEPCOVX, DEEPCOVCT DeepCovX (Sensitivity 100%) DeepCovCT(Sensitivity 97.06%) Muljo [ 64 ],(2022) Detection CXR 133,280 DENSENET121 DenseNet121(AUC average of 82.16, best AUC 99.99%) Panwar et al [ 65 ],(2021) Classification CXR 4563 CNN, ALEXNET CNN (Accuracy 98%) Nasser et al [ 66 ],(2021) Detection CXR 6000 RESNET50 ResNet50(Sensitivity 97.3%) …”

Deep Neural Network Augments Performance of Junior Residents in Diagnosing COVID-19 Pneumonia on Chest Radiographs

“…For instance, Jordan et al collected a CXR dataset from NCID to train an AI model with the DenseNet as backbone to detect COVID-19 pneumonia. In response to the rapidly evolving global pandemic during COVID-19, they quickly deployed the trained model to NCID [ 22 ]. Abul et al adopted a convolutional neural network (CNN) to extract feature representations from CXRs, and connected it with various classifiers, such as support vector machine (SVM), pattern recognition network (PRN), decision tree (DT), random forest (RF), and k-nearest neighbours (KNN), to perform COVID-19 detection [ 23 ].…”