There and Back Again: Revisiting Backpropagation Saliency Methods

Rebuffi, Sylvestre-Alvise; Fong, Ruth; Ji, Xu; Vedaldi, Andrea

doi:10.1109/cvpr42600.2020.00886

Cited by 92 publications

(50 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In genomics, where motifs can have different effects depending on whether they occur at the center vs the flanks, and also co-bind with other motifs, the assumption of positional and contextual independence may not be valid. In fact, experiments in computer vision have reported that this aspect of GradCAM leads it to fail “anywhere besides the last convolutional layer” [27], and the GradCAM paper itself states that “localization becomes progressively worse as we move to shallower convolutional layers. This is because the later convolutional layers capture high-level semantic information and at the same time retain spatial information, while the shallower layers have smaller receptive fields and only concentrate on local features that are important for the next layers” (section 5.2 of the supplement).…”

Section: Resultsmentioning

confidence: 99%

“…4 ). This suggests that a promising future direction may be to improve motif identification by combining importance scores computed at multiple different layers, similar to the approaches taken by [11], [27] and [33]. It is also worth investigating why importance scores can become worse at highlighting key features when computed closer to the input layer.…”

Section: Discussionmentioning

confidence: 99%

“…CAM-based methods are popular in computer vision because they tend to be less noisy than computing importance scores directly at the input layer - however, to our knowledge, no work has demonstrated that CAM-based methods outperform simply computing the importance scores at the same layer that the CAM method is applied to. In fact, Rebuffi et al [27] found gradient-times-activation (which they call “linear approximation”) tends to outperform GradCAM (see Table 3 in their paper), raising questions about whether the benefit of CAM is primarily due to the layer at which the importance scores are calculated, as opposed to the postion-invariant channel-weighting approach that CAM uses. Consistent with this hypothesis, we found that DeepLIFT computed at the 1st conv layer sometimes outperforms DeepLIFT computed at the input ( Fig.…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Towards More Realistic Simulated Datasets for Benchmarking Deep Learning Models in Regulatory Genomics

Prakash

Shrikumar

Kundaje

2021

Preprint

View full text Add to dashboard Cite

Deep neural networks and support vector machines have been shown to accurately predict genomewide signals of regulatory activity from raw DNA sequences. These models are appealing in part because they can learn predictive DNA sequence features without prior assumptions. Several methods such as in-silico mutagenesis, GradCAM, DeepLIFT, Integrated Gradients and GkmExplain have been developed to reveal these learned features. However, the behavior of these methods on regulatory genomic data remains an area of active research. Although prior work has benchmarked these methods on simulated datasets with known ground-truth motifs, these simulations employed highly simplified regulatory logic that is not representative of the genome. In this work, we propose a novel pipeline for designing simulated data that comes closer to modeling the complexity of regulatory genomic DNA. We apply the pipeline to build simulated datasets based on publicly-available chromatin accessibility experiments and use these datasets to benchmark different interpretation methods based on their ability to identify ground-truth motifs. We find that a GradCAM-based method, which was reported to perform well on a more simplified dataset, does not do well on this dataset (particularly when using an architecture with shorter convolutional kernels in the first layer), and we theoretically show that this is expected based on the nature of regulatory genomic data. We also show that Integrated Gradients sometimes performs worse than gradient-times-input, likely owing to its linear interpolation path. We additionally explore the impact of user-defined settings on the interpretation methods, such as the choice of "reference"/"baseline", and identify recommended settings for genomics. Our analysis suggests several promising directions for future research on these model interpretation methods. Code and links to data are available at https://github.com/kundajelab/interpret-benchmark.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Towards More Realistic Simulated Datasets for Benchmarking Deep Learning Models in Regulatory Genomics

Prakash

Shrikumar

Kundaje

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Although gradient-based methods might not be the optimal solution for visual explanation (e.g., saturation, zero-gradient image regions, and false confidence in the output score phenomena [156]), the computational cost of Grad-CAMs is negligible when compared to other methods that require multiple network forward-passes per image [156,157]. Moreover, Grad-CAM is considered the reference method in several recent works [157][158][159][160][161].…”

Section: Making the Model Interpretablementioning

confidence: 99%

Toward Learning Trustworthily from Data Combining Privacy, Fairness, and Explainability: An Application to Face Recognition

Franco

Oneto

Navarin

et al. 2021

Entropy

View full text Add to dashboard Cite

In many decision-making scenarios, ranging from recreational activities to healthcare and policing, the use of artificial intelligence coupled with the ability to learn from historical data is becoming ubiquitous. This widespread adoption of automated systems is accompanied by the increasing concerns regarding their ethical implications. Fundamental rights, such as the ones that require the preservation of privacy, do not discriminate based on sensible attributes (e.g., gender, ethnicity, political/sexual orientation), or require one to provide an explanation for a decision, are daily undermined by the use of increasingly complex and less understandable yet more accurate learning algorithms. For this purpose, in this work, we work toward the development of systems able to ensure trustworthiness by delivering privacy, fairness, and explainability by design. In particular, we show that it is possible to simultaneously learn from data while preserving the privacy of the individuals thanks to the use of Homomorphic Encryption, ensuring fairness by learning a fair representation from the data, and ensuring explainable decisions with local and global explanations without compromising the accuracy of the final models. We test our approach on a widespread but still controversial application, namely face recognition, using the recent FairFace dataset to prove the validity of our approach.

show abstract

“…For post-hoc interpretation, the prediction results are obtained by a black-box model while the interpretation is obtained separately to explain the predictive mechanism of the black box. Among the different post-hoc interpretation techniques, backpropagation methods (Zhou et al, 2016;Selvaraju et al, 2017;Wang et al, 2020a;Shrikumar et al, 2017;Rebuffi et al, 2020;Bach et al, 2015), trace from the output back to the input to determine how the different elements in the input contribute to the prediction result. Class Activation Mapping (CAM) (Zhou et al, 2016) visualizes the feature importance in convolutional neural networks by mapping the weights in the last fully connected layer to the input layer via upsampling.…”

Section: Related Workmentioning

confidence: 99%

Self-Interpretable Model with TransformationEquivariant Interpretation

Wang¹,

Wang²

2021

Preprint

View full text Add to dashboard Cite

With the proliferation of machine learning applications in the real world, the demand for explaining machine learning predictions continues to grow especially in high-stakes fields. Recent studies have found that interpretation methods can be sensitive and unreliable, where the interpretations can be disturbed by perturbations or transformations of input data. To address this issue, we propose to learn robust interpretations through transformation equivariant regularization in a self-interpretable model. The resulting model is capable of capturing valid interpretations that are equivariant to geometric transformations. Moreover, since our model is self-interpretable, it enables faithful interpretations that reflect the true predictive mechanism. Unlike existing self-interpretable models, which usually sacrifice expressive power for the sake of interpretation quality, our model preserves the high expressive capability comparable to the state-of-the-art deep learning models in complex tasks, while providing visualizable and faithful highquality interpretation. We compare with various related methods and validate the interpretation quality and consistency of our model.

show abstract

There and Back Again: Revisiting Backpropagation Saliency Methods

Cited by 92 publications

References 18 publications

Towards More Realistic Simulated Datasets for Benchmarking Deep Learning Models in Regulatory Genomics

Towards More Realistic Simulated Datasets for Benchmarking Deep Learning Models in Regulatory Genomics

Toward Learning Trustworthily from Data Combining Privacy, Fairness, and Explainability: An Application to Face Recognition

Self-Interpretable Model with TransformationEquivariant Interpretation

Contact Info

Product

Resources

About