Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging

Abstract

Saliency maps have become a widely used method to make deep learning models more interpretable by providing post-hoc explanations of classifiers through identification of the most pertinent areas of the input medical image. They are increasingly being used in medical imaging to provide clinically plausible explanations for the decisions the neural network makes. However, the utility and robustness of these visualization maps has not yet been rigorously examined in the context of medical imaging. We posit that trustworthiness in this context requires 1) localization utility, 2) sensitivity to model weight randomization, 3) repeatability, and 4) reproducibility. Using the localization information available in two large public radiology datasets, we quantify the performance of eight commonly used saliency map approaches for the above criteria using area under the precision-recall curves (AUPRC) and structural similarity index (SSIM), comparing their performance to various baseline measures. Using our framework to quantify the trustworthiness of saliency maps, we show that all eight saliency map techniques fail at least one of the criteria and are, in most cases, less trustworthy when compared to the baselines. We suggest that their usage in the high-risk domain of medical imaging warrants additional scrutiny and recommend that detection or segmentation models be used if localization is the desired output of the network.

Benchmarking saliency methods for chest X-ray interpretation

“…Furthermore, whereas many people might agree on how to segment, say, a cat or a stop sign in traditional computer vision tasks, radiologists use a certain amount of clinical discretion when defining the boundaries of a pathology on a CXR. There can also be institutional and geographic differences in how radiologists are taught to recognize pathologies, and studies have shown that there can be high interobserver variability in the interpretation of CXRs [36][37][38] . We sought to address this with the hit rate evaluation metric, which highlights when two radiologists share the same diagnostic intention, even if it is less exact than IoU in comparing segmentations directly.…”

“…The copyright holder for this preprint this version posted March 2, 2021. ; https://doi.org/10.1101/2021.02.28.21252634 doi: medRxiv preprint Our work builds upon several studies investigating the validity of saliency maps in localization 39,40 and upon some early work on trustworthiness of saliency methods to explain DNNs in medical imaging 41 . We substantially extend the body of literature by doing a comprehensive analysis on a multi-label classification task using the most popular Table 2 for dataset summary statistics.…”

“…Our work builds upon several studies investigating the validity of saliency maps in localization 39,40 and upon some early work on trustworthiness of saliency methods to explain DNNs in medical imaging 41 . We substantially extend the body of literature by doing a comprehensive analysis on a multi-label classification task using the most popular saliency method in the medical context, Grad-CAM.…”

Machine Learning Solutions for Osteoporosis—A Review

“…( 104 ) To assess the reliability of heat maps on images, Arun and colleagues explored their capability in localizing region of interest in medical imaging DL models. ( 129 ) None of the explanatory methods tested passed all validity checks, so a careful interpretation of these visualization tools is required. Simplified illustrations of the sophisticated ML solutions will help clinicians evaluate their actual potential for incorporation into routine clinical practice and dispel the “black box” perception of AI; hence, such illustrative approaches should be encouraged and ideally included in the standard checklist for ML studies.…”

AI for radiographic COVID-19 detection selects shortcuts over signal

“…While saliency maps are widely used to interpret image-based artificial intelligence systems [32,33,46], the reliability of these approaches has been disputed by contemporary work, which observes that saliency maps explaining medical imaging classifiers fail to localize medically relevant pathology [47]. However, this prior work did not disentangle whether (i) the saliency maps fail to identify the features that are important for the classification models, or (ii) the saliency maps faithfully identify the features that are important for the classification models, but the models do not depend on medically relevant pathology.…”