Improving Uncertainty Estimation With Semi-Supervised Deep Learning for COVID-19 Detection Using Chest X-Ray Images

Abstract

In this work we implement a COVID-19 infection detection system based on chest Xray images with uncertainty estimation. Uncertainty estimation is vital for safe usage of computer aided diagnosis tools in medical applications. Model estimations with high uncertainty should be carefully analyzed by a trained radiologist. We aim to improve uncertainty estimations using unlabelled data through the MixMatch semi-supervised framework. We test popular uncertainty estimation approaches, comprising Softmax scores, Monte-Carlo dropout and deterministic uncertainty quantification. To compare the reliability of the uncertainty estimates, we propose the usage of the Jensen-Shannon distance between the uncertainty distributions of correct and incorrect estimations. This metric is statistically relevant, unlike most previously used metrics, which often ignore the distribution of the uncertainty estimations. Our test results show a significant improvement in uncertainty estimates when using unlabelled data. The best results are obtained with the use of the Monte Carlo dropout method.

The Applications of Artificial Intelligence in Chest Imaging of COVID-19 Patients: A Literature Review

“…Furthermore, other authors advised the sampling of large datasets to reduce predictive uncertainty, even though most works used small image samples, due to the lack of large open COVID-19 datasets (particularly for CXR) [139][140][141][142]. This is why further studies are needed to implement AI capacities in the above discussed settings (identification, screening, patients' stratification and differential diagnosis), in order to guide the development of AI-empowered tools to reduce human error and assist radiologists in their decision-making process.…”

Uncertainty-aware convolutional neural network for COVID-19 X-ray images classification

“…Furthermore, they have applied various machine learning and statistical modeling methods for COVID-19 diagnosis and measuring the epistemic uncertainty of classification results. Calderon-Ramirez et al [ 45 ] have proposed MixMatch semi-supervised framework for improving uncertainty estimation of model for COVID-19 detection. They have tested uncertainty estimation techniques like softmax scors, deterministic uncertainty quantification and MC dropout.…”

“…The majority of existing approaches [ [28] , [29] , [30] , [31] , [32] , [33] , [34] , [35] , [36] , [37] , [38] , [39] , [40] , [41] , [42] , [43] ] have not provided information about the uncertainty associated with their predictions. Only a few studies [ [44] , [45] , [46] ] have provided their model uncertainty information. Whereas an uncertainty estimation of the model is very important in the medical domain for a reliable and safe CAD system.…”

Machine Learning for Health: Algorithm Auditing & Quality Control

“…Model performance estimates are often not valid for the types of varying input distribution that can occur during real world deployment [ 15 – 17 ]. The decision heuristics a model learns can differ from the heuristics we may expect a human to use [ 1 , 18 – 20 ], and model predictions may come with ill-calibrated statements of confidence [ 21 – 23 ] or no estimate of uncertainty altogether [ 24 ]. Developers proposing new ML4H technologies sometimes promise to match or even surpass the performance of existing methods [ 25 ] yet the reality is often more complicated.…”