Neural Response Interpretation through the Lens of Critical Pathways

Khakzar, Ashkan; Baselizadeh, Soroosh; Khanduja, Saurabh; Rupprecht, Christian; Kim, Seong-Tae; Navab, Nassir

doi:10.1109/cvpr46437.2021.01332

Cited by 22 publications

(17 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…FullGrad leverages the activation, gradient and, bias values from all layers. PathwayGrad [13] leverages critical pathways (pathway important neurons).…”

Section: Explaining Predictions Via Feature Attributionmentioning

confidence: 99%

“…The problem is called feature attribution [16,35], and the solutions are commonly known as explanation, attribution, or saliency methods. There is an extensive list of explanation methods in the literature [8,9,13,15,16,19,26,27,29,32,33,35,39,41]. One peculiar observation is that these solutions point to different features as being important.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Do Explanations Explain? Model Knows Best

Khakzar¹,

Khorsandi²,

Nobahari³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

It is a mystery which input features contribute to a neural network's output. Various explanation (feature attribution) methods are proposed in the literature to shed light on the problem. One peculiar observation is that these explanations (attributions) point to different features as being important. The phenomenon raises the question, which explanation to trust? We propose a framework for evaluating the explanations using the neural network model itself. The framework leverages the network to generate input features that impose a particular behavior on the output. Using the generated features, we devise controlled experimental setups to evaluate whether an explanation method conforms to an axiom. Thus we propose an empirical framework for axiomatic evaluation of explanation methods. We evaluate well-known and promising explanation solutions using the proposed framework. The framework provides a toolset to reveal properties and drawbacks within existing and future explanation solutions. 1

show abstract

“…FullGrad leverages the activation, gradient and, bias values from all layers. PathwayGrad [13] leverages critical pathways (pathway important neurons).…”

Section: Explaining Predictions Via Feature Attributionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Do Explanations Explain? Model Knows Best

Khakzar¹,

Khorsandi²,

Nobahari³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Others address this problem by multi-task learning [10,26] that used selective instances for training on imbalanced sets for each task. Interpreting Neural Networks: Two principal neural network interpretation approaches are feature attribution [29,32,13,30,22,17,12] (i.e. saliency methods [4]) and analyzing internal units (e.g.…”

Section: Related Workmentioning

confidence: 99%

Analyzing the Effects of Handling Data Imbalance on Learned Features from Medical Images by Looking Into the Models

Khakzar¹,

Li²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

One challenging property lurking in medical datasets is the imbalanced data distribution, where the frequency of the samples between the different classes is not balanced. Training a model on an imbalanced dataset can introduce unique challenges to the learning problem where a model is biased towards the highly frequent class. Many methods are proposed to tackle the distributional differences and the imbalanced problem. However, the impact of these approaches on the learned features is not well studied. In this paper, we look deeper into the internal units of neural networks to observe how handling data imbalance affects the learned features. We study several popular cost-sensitive approaches for handling data imbalance and analyze the feature maps of the convolutional neural networks from multiple perspectives: analyzing the alignment of salient features with pathologies and analyzing the pathology-related concepts encoded by the networks. Our study reveals differences and insights regarding the trained models that are not reflected by quantitative metrics such as AUROC and AP and show up only by looking at the models through a lens.

show abstract

“…For post-hoc explanation of chest X-ray models, many works opt for feature attribution methodologies [21,23,1,18,9,25]. These works use feature attribution methods, such as Class Activation Maps (CAM) [25,18] to reveal which input regions are contributing to the output prediction.…”

Section: Introductionmentioning

confidence: 99%

Towards Semantic Interpretation of Thoracic Disease and COVID-19 Diagnosis Models

Khakzar¹,

Musatian²,

Buchberger³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Convolutional neural networks are showing promise in the automatic diagnosis of thoracic pathologies on chest x-rays. Their blackbox nature has sparked many recent works to explain the prediction via input feature attribution methods (aka saliency methods). However, input feature attribution methods merely identify the importance of input regions for the prediction and lack semantic interpretation of model behavior. In this work, we first identify the semantics associated with internal units (feature maps) of the network. We proceed to investigate the following questions; Does a regression model that is only trained with COVID-19 severity scores implicitly learn visual patterns associated with thoracic pathologies? Does a network that is trained on weakly labeled data (e.g. healthy, unhealthy) implicitly learn pathologies? Moreover, we investigate the effect of pretraining and data imbalance on the interpretability of learned features. In addition to the analysis, we propose semantic attribution to semantically explain each prediction. We present our findings using publicly available chest pathologies (CheXpert [5], NIH ) and , and COVID-19 chest X-ray segmentation dataset [4]). The Code 1 is publicly available.

show abstract

Neural Response Interpretation through the Lens of Critical Pathways

Cited by 22 publications

References 24 publications

Do Explanations Explain? Model Knows Best

Do Explanations Explain? Model Knows Best

Analyzing the Effects of Handling Data Imbalance on Learned Features from Medical Images by Looking Into the Models

Towards Semantic Interpretation of Thoracic Disease and COVID-19 Diagnosis Models

Contact Info

Product

Resources

About