How explainable are adversarially-robust CNNs?

Abstract

Three important criteria of existing convolutional neural networks (CNNs) are (1) test-set accuracy; (2) out-of-distribution accuracy; and (3) explainability. While these criteria have been studied independently, their relationship is unknown. For example, do CNNs that have a stronger out-of-distribution performance have also stronger explainability? Furthermore, most prior feature-importance studies only evaluate methods on 2-3 common vanilla ImageNet-trained CNNs, leaving it unknown how these methods generalize to CNNs of other architectures and training algorithms. Here, we perform the first, large-scale evaluation of the relations of the three criteria using 9 feature-importance methods and 12 ImageNet-trained CNNs that are of 3 training algorithms and 5 CNN architectures. We find several important insights and recommendations for ML practitioners. First, adversarially robust CNNs have a higher explainability score on gradient-based attribution methods (but not CAM-based or perturbation-based methods). Second, AdvProp models, despite being highly accurate more than both vanilla and robust models alone, are not superior in explainability. Third, among 9 feature attribution methods tested, GradCAM and RISE are consistently the best methods. Fourth, Insertion and Deletion are biased towards vanilla and robust models respectively, due to their strong correlation with the confidence score distributions of a CNN. Fifth, we did not find a single CNN to be the best in all three criteria, which interestingly suggests that CNNs are harder to interpret as they become more accurate.Preprint. Under review.

A Quality Aware Sample-to-Sample Comparison for Face Recognition

“…Unlike triplet and contrastive losses, softmax-based losses are not subject to explicit easy/hard sample mining [42,31]. In this section, using a simple toy example, we show that the Softmax-based losses implicitly benefit easy/hard sample mining by their gradient.…”

Rethinking Robustness of Model Attributions

“…In other words, these differently trained model guard the sample image against the attributional top-k attack. Recent work by Nourelahi et al (2022) has empirically studied the effectiveness of adversarially (PGD) trained models in obtaining better attributions, e.g., Figure 7(center) shows sharper attributions to features highlighting the ground-truth class.…”

On the Robustness of Explanations of Deep Neural Network Models: A Survey

“…The key concerns related to attributions that we discussed in earlier sections are that: (1) the attribution maps have to be of good quality, and (2) they should be robust under perturbations. It is observed that adversarially robust models obtain the best explanation maps under different attribution methods (Figure 10 from Nourelahi et al [82]) as they seem to be more focused on the objects related to the input's ground-truth class. This section explores the interplay between explanations and adversarial robustness of neural network models, primarily from an image input perspective where adversarial robustness is more studied.…”