Improving Adversarial Robustness via Probabilistically Compact Loss with Logit Constraints

Abstract

Convolutional neural networks (CNNs) have achieved state-of-the-art performance on various tasks in computer vision. However, recent studies demonstrate that these models are vulnerable to carefully crafted adversarial samples and suffer from a significant performance drop when predicting them. Many methods have been proposed to improve adversarial robustness (e.g., adversarial training and new loss functions to learn adversarially robust feature representations). Here we offer a unique insight into the predictive behavior of CNNs that they tend to misclassify adversarial samples into the most probable false classes. This inspires us to propose a new Probabilistically Compact (PC) loss with logit constraints which can be used as a drop-in replacement for cross-entropy (CE) loss to improve CNN's adversarial robustness. Specifically, PC loss enlarges the probability gaps between true class and false classes meanwhile the logit constraints prevent the gaps from being melted by a small perturbation. We extensively compare our method with the state-of-the-art using large scale datasets under both white-box and black-box attacks to demonstrate its effectiveness. The source codes are available at https://github.com/xinli0928/PC-LC.

School of Mechanical Engineering, Shanghai Jiao Tong University, 200240, Shanghai, China

“…Defensive Detectors. There are a variety of ways to have a defense model, including input transformation (Guo et al, 2017), adversarial training (Pang et al, 2020), and improved loss functions (Li et al, 2021). Notice the fact that we want to compare different attack methods on a defense model, using adversarial training is not reasonable enough.…”

“…Then, we take two methods that are more often used in adversarial defense. We trained two robust detectors using Faster RCNN including the input manipulation of JPEG (Guo et al, 2017) and an improved loss function of Probabilistically Compact Loss (PC Loss) (Li et al, 2021). The input compression by JPEG is to neutralize the influence of adversarial noises and the usage of PC Loss instead of Cross Entropy Loss is mainly to enlarge the gaps of classification probabilities and therefore strengthen the robustness.…”

Improving Adversarial Robustness with Hypersphere Embedding and Angular-based Regularizations

“…Given that the angles between the feature vector and weight vectors contain abundant discriminative information [10,16,17] and adversarial attacks attack these angles, we propose a regularization term that directly encourages the weight-feature compactness, more specifically, by minimizing the angle between adversarial feature vector and the weight vector corresponding to the ground-truth label y. In addition, prior works [18] have argued strong connections between adversarial robustness and inter-class separability. We therefore propose an additional angular-based regularization term that improves the inter-class separability.…”

Learning Compact Features via In-Training Representation Alignment

“…(Ioffe and Szegedy 2015) proposes batch normalization (BN) to reduce the internal covariate shift caused by SGD. For image classification, data-augmentation types of regularization are also developed (DeVries and Taylor 2017; Gastaldi 2017; Li et al , 2021. Different from those approaches, our proposed ITRA is motivated by the perspective of exact gradient update for each mini-batch in SDG training, and achieves regularization by encouraging the alignment of feature representations of different mini-batches.…”