Parameterizing Activation Functions for Adversarial Robustness

Abstract

The bulk of existing research in defending against adversarial examples focuses on defending against a single (typically bounded p -norm) attack, but for a practical setting, machine learning (ML) models should be robust to a wide variety of attacks. In this paper, we present the first unified framework for considering multiple attacks against ML models. Our framework is able to model different levels of learner's knowledge about the test-time adversary, allowing us to model robustness against unforeseen attacks and robustness against unions of attacks. Using our framework, we present the first leaderboard, MultiRobustBench, for benchmarking multiattack evaluation which captures performance across attack types and attack strengths. We evaluate the performance of 16 defended models for robustness against a set of 9 different attack types, including p -based threat models, spatial transformations, and color changes, at 20 different attack strengths (180 attacks total). Additionally, we analyze the state of current defenses against multiple attacks. Our analysis shows that while existing defenses have made progress in terms of average robustness across the set of attacks used, robustness against the worst-case attack is still a big open problem as all existing models perform worse than random guessing.

RobArch: Designing Robust Architectures against Adversarial Attacks

“…SiLU further improves the accuracy (Table 1 row 16), which echoes the result in [57]. Recently, Dai et al [9] added learnable parameters to original non-parametric functions, and proposed the parametric counterparts, e.g., ReLU to Parametric ReLU (PReLU) and SiLU to Parametric SiLU (PSiLU) or Parametric Shifted SiLU (PSSiLU). These parametric functions outperform the non-parametric ones on CIFAR-10 [28].…”

“…A huge number of AT variants have been proposed, e.g., TRADES [64], AWP [56], ADT [15], DART [52], MART [53], CAS [4], Max-Margin AT [14], etc. For the robust DNN research, only a few studies explored how architectures affect robustness [13,36,46,48], e.g., depths [60], widths [55] and activation functions [9,57]. However, the total model capacity is unconstrained along with the architecture modifications.…”

“…The guidelines present significant new knowledge and discoveries for our computer vision community. For example, we have discovered (1) deepening a network is more effective than widening it, and there is a sweet spot; (2) specific modifications such as adding Squeeze and Excitation (SE) block, removing the first normalization layer in a block, and reducing the downsampling factor in the stem stage effectively boosts robustness; and (3) architecture designs that harm robustness include inverted bottleneck, large dilation factor, Instance Normalization (IN), parametric activation functions [9], and reducing activation layers. • Top performance against strong adversarial attacks.…”

How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses

“…Self-COnsistent Robust Error (SCORE) [121] employs local equivariance to minimize the robustness error of a network, easing the reconciliation between robustness and accuracy and dealing with worst-case uncertainty. Parametric Shifted Sigmoidal Linear Unit (PSSiLU) [122] changes the activation function to create high finite curvature and output positive results with negative inputs, creating a parameterized activation function, that is used with adversarial training.…”