Understanding Deep Networks via Extremal Perturbations and Smooth Masks

Abstract

The problem of attribution is concerned with identifying the parts of an input that are responsible for a model's output. An important family of attribution methods is based on measuring the effect of perturbations applied to the input. In this paper, we discuss some of the shortcomings of existing approaches to perturbation analysis and address them by introducing the concept of extremal perturbations, which are theoretically grounded and interpretable. We also introduce a number of technical innovations to compute extremal perturbations, including a new area constraint and a parametric family of smooth perturbations, which allow us to remove all tunable hyper-parameters from the optimization problem. We analyze the effect of perturbations as a function of their area, demonstrating excellent sensitivity to the spatial properties of the deep neural network under stimulation. We also extend perturbation analysis to the intermediate layers of a network. This application allows us to identify the salient channels necessary for classification, which, when visualized using feature inversion, can be used to elucidate model behavior. Lastly, we introduce TorchRay 1 , an interpretability library built on PyTorch.

Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications

“…A present trend in the ML community is a migration to the PyTorch framework with its eager execution paradigm, away from other back ends. Both the TorchRay [46] and Captum [95] packages for Python and PyTorch enable the use of interpretability methods for neural network models defined in the context of PyTorch's high-level neural network description modules. Captum can be understood as a rich selection of XAI methods based on modified backprop and is part of the PyTorch project itself.…”

Interpreting Neural Networks for Biological Sequences by Learning Stochastic Masks

“…Specifically, using APARENT (Bogard et al, 2019) -a CNN capable of predicting cleavage and polyadenylation from primary sequence -we trained an Inclusion-Scrambler to reconstruct isoform predictions for the original APARENT training data (t bits = 0.25, λ = 1; Figure 3A). As we anticipated important polyadenylation features like RNA binding protein (RBP) recognition motifs to consist of short subsequences, we regularized the Scrambler by fixing the final layer of the network to a Gaussian filter (reminiscent of a technique proposed by Fong et al, 2019) to encourage the selection of contiguous nucleotides for masking. We found that the regularized Scrambler learned to recognize known regulatory binding factors associated with alternative polyadenylation (Supplementary Fig.…”

Explaining Image Classifiers Using Statistical Fault Localization

“…In RISE [25], the importance of a pixel is computed as the expectation over all local perturbations conditioned on the event that the pixel is observed. More recently, the concept of "extreme perturbations" has been introduced to improve the perturbation analysis by the extremal algorithm [6].…”

“…We now present the experimental results. We tested DeepCover on a variety of DNN models for ImageNet and we compare DeepCover with the most popular and most recent work in AI explanation: lime [27], shap [18], gradcam [29], rise [25] and extremal [6]. 6…”

“…The DNN is VGG16 and we randomly sample 1,000 images from ILSVRC2012 as inputs. We evaluate the effect of highly ranked features by different methods following an addition/deletion style experiment [25,6].…”