The Multimodal Deep Learning for Diagnosing COVID-19 Pneumonia from Chest CT-Scan and X-Ray Images

Abstract

Due to the COVID-19 Pandemic, doctors need to make medical decisions for their patients based on many examinations (e.g., polymerase chain reaction test, temperature test, CT-Scans, or X-rays). However, transfer learning has been used in several researches and focuses on only a single modality of biomarkers (e.g., CT-Scan or X-Ray) for diagnosing Pneumonia. In recent studies, a single modality has its own classification accuracy and every different biomarker may provide complementary information for detecting COVID-19 Pneumonia. The COVID-19 virus can be detected by CT-Scan and X-Ray imaging of the chest. In this work, we propose to use concatenation of two different transfer learning models using an open-source dataset of 2500 CT-Scan images and 2500 X-ray images for classifying CT-Scan images and X-ray images into two classes: normal and COVID-19 Pneumonia. We have used DenseNet121, MobileNet, Xception, InceptionV3, ResNet50, and VGG16 models for image recognition in our work. As a result, we achieve the best classification accuracy of 99.87% of the concatenation of ResNet50 and VGG16 networks. We also achieved the best classification accuracy of 98.00% when using a single modality of CT-Scan ResNet50 networks and classification accuracy of 98.93% for X-Ray VGG16 networks. Our multimodal fusion method shows a better classification accuracy compared to the method of using a single modality of biomarkers.

Machine learning empowered COVID-19 patient monitoring using non-contact sensing: An extensive review

“…X-ray images can be used to detect and monitor the symptoms of patients with COVID-19 ( Fig. 5 ) [ 100 ]. For instance, X-ray images of the lungs are used to detect abnormal respiration.…”

“…5 Sample images of normal person and patients with COVID-19 (left) and histograms of the images (right) (reprinted from Ref. [ 100 ] with permission). …”

“…The final outcome of the images reveals any types of abnormalities that can eventually be utilized to detect symptoms of COVID-19 ( Fig. 5 ) [ 100 , [105] , [106] , [107] ]. Research has shown that CT scanning has an exposure sensitivity of 86%–98% and can hence monitor any activity of SARS-CoV-2 infection in the lungs [ 108 ].…”

COVID-19 Detection Based on Image Regrouping and Resnet-SVM Using Chest X-Ray Images

“…where t g denotes the projection of updating parameters and t v denotes the exponential average of squares gradients. The stride can de described in (16).…”

“…Experimental results showed that with the limited training data, most of the deeper networks struggled to train well and provided less consistency. Hilmizen et al [16] proposed a combined two different transfer learning models method for CT-Scan and CXR images classification. The authors collected 2,500 CT images and 2,500 CXR images into two classes of normal and COVID-19, in this work, they used Densenet-121, Mobile net, X-ception, Inception-v3, ResNet-50 and VGG-16 for classification.…”