On Data Bias and the Usability of Deep Learning Algorithms in Classifying COVID-19 based on Chest X-ray

Abstract

SARS-COV-2 is a new strain of virus that was first detected in China. It quickly spread across the world affecting millions of people. For this reason, early detection of the virus is mandatory in order to limit the spread of the virus. Real-time reverse transcription polymerase chain reaction (RT-PCR) and the antibody test are the main tests used to detect the virus. Chest X-rays (CXRs) and computerized tomography (CT) scans are also used to detect the virus although the American college of Radiology does not recommend using medical imaging as a diagnostic tool. Like other medical imaging, convolutional neural networks are used to classify the images. We believe that developing a model to detect COVID-19 has no clinical value regardless of the accuracy achieved since 58% of CXRs seem to be normal. During literature review, several papers with suspicious accuracy of 90% and higher were found. We believe that the dataset used to train and validate the network is biased and is not appropriate for deep learning as any model we train using the same dataset has achieved high accuracy. Our experiments on Cohen's Covid dataset, augmented with Wang dataset, shows that any model trained on Cohen dataset can easily achieve high accuracy. This was further validated with two experienced radiologists who participated in this study were only able to classify 60% as being Covid. Our study highlight the importance of addressing bias in data and developing trustworthy and explainable ML models based on well curated data.

Pathological changes or technical artefacts? The problem of the heterogenous databases in COVID-19 CXR image analysis

“…The closest confounder found in the literature to the resolution artefact is so-called the "picture of picture" issue and lack of access to the original DICOM data. Those problems were mentioned in articles [8] , [9] , [21] but without detailed analysis. We are filling this research gap with our study by showing how the low resolution of the input images can adversely affect network generalization.…”

“…The perfect results of COVID-19 prediction models reported by many scientists were too good to be true [9] . The deciding factor was the data used in each research; the model worked exceptionally well on the test set but lacked generalization to work on data outside the research scope [10] .…”

“…The lack of access to medical data turned researchers to other sources. They began to use radiopaedia.org and other sites providing images of unknown origin saved in a standard image format with unknown compression ratios [8] , [9] , [18] , [21] , [22] , [23] . So-called Frankenstein datasets were created by stitching together " picture-of-a-picture " data of the originals [21] .…”