Parameterizing Activation Functions for Adversarial Robustness

Abstract

Deep neural networks are known to be vulnerable to adversarially perturbed inputs. A commonly used defense is adversarial training, whose performance is influenced by model capacity. While previous works have studied the impact of varying model width and depth on robustness, the impact of increasing capacity by using learnable parametric activation functions (PAFs) has not been studied. We study how using learnable PAFs can improve robustness in conjunction with adversarial training. We first ask the question: how should we incorporate parameters into activation functions to improve robustness? To address this, we analyze the direct impact of activation shape on robustness through PAFs and observe that activation shapes with positive outputs on negative inputs and with high finite curvature can increase robustness. We combine these properties to create a new PAF, which we call Parametric Shifted Sigmoidal Linear Unit (PSSiLU). We then combine PAFs (including PReLU, PSoftplus and PSSiLU) with adversarial training and analyze robust performance. We find that PAFs optimize towards activation shape properties found to directly affect robustness. Additionally, we find that while introducing only 1-2 learnable parameters into the network, smooth PAFs can significantly increase robustness over ReLU. For instance, when trained on CIFAR-10 with additional synthetic data, PSSiLU improves robust accuracy by 4.54% over ReLU on ResNet-18 and 2.69% over ReLU on WRN-28-10 in the ∞ threat model while adding only 2 additional parameters into the network architecture. The PSSiLU WRN-28-10 model achieves 61.96% AutoAttack accuracy, improving over the state-of-the-art robust accuracy on RobustBench . Overall, our work puts into context the importance of activation functions in adversarially trained models.

Collaborative Adversarial Training

“…Rebuffi et al [32] focus on data augmentation and study the performance of using generative models. There are also insightful work that focuses on model architectures [18,41,3,26,30], batch normalization [43], and activation functions [44,11]. Distilling from adversarially trained models has also been widely studied [49,35,2].…”

Understanding Adversarial Robustness from Feature Maps of Convolutional Layers

“…Recently, Dai et al [6] studied the learnable parametric activation functions and the developed PSSiLU can significantly increase robustness when extra training samples are available. Shao et al [29] showed that vanilla ViTs [7] can learn more generalizable features and thus has superior robustness against adversarial perturbations.…”

Adversarial Attack on Attackers: Post-Process to Mitigate Black-Box Score-Based Query Attacks

“…We evaluate AAA along with 8 defense baselines, including random noise defense (RND [33]), adversarial training (AT [36,59,49]), dynamic inference (DENT [19]), training randomness (PNI [51]), and ensemble (TRS [60]). Results of AT with extra data [22]) are in Appendix A. AAA uses hyper-parameters α = 1, t = 6, β = 5, and an Adam optimizer [61] with learning rate 0.1, β 1 = 0.9, β 2 = 0.999, and the iteration of optimization 100.…”

“…3 (left). AAA, indicated by the orange lines, changes the scores slightly without hurting accuracy (dotted lines) compared to RND [33] and AT [36]. By such small modifications, AAA not only improves the calibration but also prevents SQAs by misleading them into incorrect directions.…”

“…By post-processing, AAA not only lowers the calibration error without hurting accuracy in all cases, but also efficiently prevents SQAs, e.g., helping a WideResNet-28 [34] on CIFAR-10 [35] secure 80.59% accuracy under Square attack [5] (2500 queries), while the best prior defense (i.e., adversarial training [36]) only attains 67.44%. Because AAA attacks the general greedy update of SQAs, such advantages of AAA over 8 baseline defenses can be consistently observed on 8 CIFAR-10/ImageNet models under 6 SQAs, using different attack targets, norms, bounds, and losses, verifying AAA a user-friendly, effective, and generalizable defense.…”