Evaluating the Visualization of What a Deep Neural Network Has Learned

Abstract

Abstract-Deep Neural Networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multi-layer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the "importance" of individual pixels wrt the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012 and MIT Places data sets. Our main result is that the recently proposed Layer-wise Relevance Propagation (LRP) algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of neural network performance.

Interpretability of deep learning models: A survey of results

“…This makes the method particularly well suited to analyzing image classifiers, though the method has also been adapted for text and electroencephalogram signal classification [31]. Samek et al [32] have also developed an objective metric for comparing the output of LRP with similar heatmapping algorithms. Kumar et al [33] present an alternative heat-mapping method that can show the image regions that the model was most attentive to, but also allows for multiple classes to be associated with these regions of attention, whereas LRP assumes all features make either a zero or positive contribution to the single predicted class.…”

“…This means that explanations of the same type can be compared using a metric without need for any further context [32]. However, explanations of different types (saliency map images [13] and text captions for example [22]) can't be compared using a metric.…”

Visualization of feature evolution during convolutional neural network training

“…Simonyan et al [11] compare this method with a form of activation maximization. In [12], the authors show sensitivity maps with evidence both for and against a particular class, while [13] develops heatmaps showing relevance or importance of image regions.…”

Interpretable deep neural networks for single-trial EEG classification

“…It is a principled method which has close relation to Taylor decomposition [11] and is applicable to arbitrary DNN architectures. From a practitioners perspective LRP adds a new dimension to the application of DNNs (e.g., in computer vision [12], [13]) by making the prediction transparent. Within the scope of cognitive neuroscience this means that DNN with LRP, may provide not only a highly effective (non-linear) classification technique that is suitable for complex high-dimensional data, but also yield detailed single-trial accounts of the distribution of decision-relevant information, a feature that is lacking in commonly applied DNN techniques and also in other state-of-the art methods (such as those discussed below).…”