Concise Explanations of Neural Networks using Adversarial Training

Abstract

We show new connections between adversarial learning and explainability. One form of explanation of the output of a neural network model in terms of its input features, is a vector of feature-attributions using the Integrated Gradient (IG) method. Two desirable characteristics of an attribution-based explanation are:(1) sparseness: the attributions of irrelevant or weakly relevant features should be negligible, thus resulting in concise explanations in terms of the significant features, and (2) stability: it should not vary significantly within a small local neighborhood of the input. For a general class of convex loss functions, we show two new theoretical results: (a) via a novel analysis of the Stochastic Gradient Descent process, we show that adversarial training using an ∞ -bounded adversary produces sparse models (and hence concise explanations), and (b) we show that natural model-training while encouraging stable IG-explanations (via an extra term in the loss function), is equivalent to adversarial training. We demonstrate the first phenomenon in experiments training logistic regression models on a variety of tabular toy and real-world advertising datasets: adversarial training yields significantly sparser models, with little or no degradation in performance on natural test data.Preprint. Under review.

XAIR: A Systematic Metareview of Explainable AI (XAI) Aligned to the Software Development Process

“…At their core, they rely on the importance distribution onto the entire set of features. For example, Chalasani et al [178] use the Gini-index of the feature attribution vector as a measurement of its sparseness. The sparser the vector, the lower is the complexity of the explanation, since fewer features are required to explain the model behavior.…”