Nuances of Interpreting X-ray Analysis by Deep Learning and Lessons for Reporting Experimental Findings

Abstract

With the increase in the availability of annotated X-ray image data, there has been an accompanying and consequent increase in research on machine-learning-based, and ion particular deep-learning-based, X-ray image analysis. A major problem with this body of work lies in how newly proposed algorithms are evaluated. Usually, comparative analysis is reduced to the presentation of a single metric, often the area under the receiver operating characteristic curve (AUROC), which does not provide much clinical value or insight and thus fails to communicate the applicability of proposed models. In the present paper, we address this limitation of previous work by presenting a thorough analysis of a state-of-the-art learning approach and hence illuminate various weaknesses of similar algorithms in the literature, which have not yet been fully acknowledged and appreciated. Our analysis was performed on the ChestX-ray14 dataset, which has 14 lung disease labels and metainfo such as patient age, gender, and the relative X-ray direction. We examined the diagnostic significance of different metrics used in the literature including those proposed by the International Medical Device Regulators Forum, and present the qualitative assessment of the spatial information learned by the model. We show that models that have very similar AUROCs can exhibit widely differing clinical applicability. As a result, our work demonstrates the importance of detailed reporting and analysis of the performance of machine-learning approaches in this field, which is crucial both for progress in the field and the adoption of such models in practice.

Extracting and Classifying Salient Fields of View from Microscopy Slides of Tuberculosis Bacteria

“…The same expert manually created binary masks corresponding to all of the 130 FOVs, which were treated as the ground truth. In order to facilitate a comprehensive and nuanced comparison between models [35], we assess performance using a number of metrics, namely overall accuracy, recall (sensitivity) and precision (specificity), receiver operating characteristics (ROC), and the area under the ROC curve (AUC).…”

Research and implementation of multi-disease diagnosis on chest X-ray based on vision transformer

“…Additionally, we introduced an attention region selection module to guide the model’s focus toward crucial areas of the image. The proposed model had an average area under the curve (AUC) of 0.837 for diagnosing 14 diseases in the ChestX-ray14 data set, representing a 2.1% improvement compared to that of previous models ( 2 - 5 ). Notably, the most significant enhancements were observed for pneumonia and edema diseases, with increases of 7.5% and 6.8%, respectively.…”