On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation

Abstract

Understanding and interpreting classification decisions of automated image classification systems is of high value in many applications, as it allows to verify the reasoning of the system and provides additional information to the human expert. Although machine learning methods are solving very successfully a plethora of tasks, they have in most cases the disadvantage of acting as a black box, not providing any information about what made them arrive at a particular decision. This work proposes a general solution to the problem of understanding classification decisions by pixel-wise decomposition of nonlinear classifiers. We introduce a methodology that allows to visualize the contributions of single pixels to predictions for kernel-based classifiers over Bag of Words features and for multilayered neural networks. These pixel contributions can be visualized as heatmaps and are provided to a human expert who can intuitively not only verify the validity of the classification decision, but also focus further analysis on regions of potential interest. We evaluate our method for classifiers trained on PASCAL VOC 2009 images, synthetic image data containing geometric shapes, the MNIST handwritten digits data set and for the pre-trained ImageNet model available as part of the Caffe open source package.

Visualizing and Understanding Neural Machine Translation

“…Our work is closely related to previous visualization approaches that compute the contribution of a unit at the input layer to the final decision at the output layer (Simonyan et al, 2014;Mahendran and Vedaldi, 2015;Nguyen et al, 2015;Girshick et al, 2014;Bach et al, 2015;. Among them, our approach bears most resemblance to (Bach et al, 2015) since we adapt layer-wise relevance propagation to neural machine translation.…”

“…The contextual word set of a hidden state v ∈ R M ×1 is denoted as C(v), which is a set of source and target contextual word vectors u ∈ R N ×1 that influences the generation of v. (Bach et al, 2015).…”

“…We follow (Bach et al, 2015) to use layer-wise relevance propagation (LRP) to compute neuronlevel relevance. We use a simple feed-forward network shown in Figure 3 to illustrate the central idea of LRP.…”

“…Note that we ignore the non-linear activation function because Bach et al (2015) indicate that LRP is invariant against the choice of non-linear function. Then, the relevance is further propagated to the input layer:…”

Interpretable deep neural networks for single-trial EEG classification

“…This redistribution rule has been showed to fulfill the layer-wise conservation property [10] and to be closely related to a deep variant of Taylor decomposition [11].…”