Subspace Adversarial Training

Li, Tao; Wu, Yingwen; Chen, Sizhe; Fang, Kun; Huang, Xiaolin

doi:10.1109/cvpr52688.2022.01305

Cited by 44 publications

(6 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This phenomenon is defined as the classification accuracy increasing abruptly on the mixed dataset containing clean and adversarial examples but dropping sharply on the dataset containing only clean examples. Li et al [46] believed that the phenomenon of robust accuracy overfitting was due to the sudden increase in the gradient. By carrying out the FGSM attack in the low circumferential space of several model checkpoints, they limited the gradient norm to a range of smooth variation, alleviating the overfitting phenomenon while achieving comparable results with the iterative attack.…”

Section: • Fgsm-related Methodsmentioning

confidence: 99%

Generation and Countermeasures of Adversarial Examples on Vision: A Survey

Liu,

Li,

Guo

et al. 2023

Preprint

View full text Add to dashboard Cite

Recent studies have found that deep learning models are vulnerable to adversarial examples, demonstrating that applying a certain imperceptible perturbation on clean examples can effectively deceive the well-trained and high-accuracy deep learning models. Moreover, the adversarial examples can achieve a considerable level of certainty with the attacked label. In contrast, human could barely discern the difference between clean and adversarial examples, which raised tremendous concern about robust and trustworthy deep learning techniques. In this survey, we reviewed the existence, generation, and countermeasures of adversarial examples in Computer Vision, to provide comprehensive coverage of the field with an intuitive understanding of the mechanisms and summarized the strengths, weaknesses, and major challenges. We hope this effort will ignite further interest in the community to solve current challenges and explore this fundamental area.

show abstract

Section: • Fgsm-related Methodsmentioning

confidence: 99%

Generation and Countermeasures of Adversarial Examples on Vision: A Survey

Liu,

Li,

Guo

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…It not only adopts Fast Gradient Sign Method (FGSM) [17] to generate adversarial samples during the training but also incorporates a cyclic learning rate [43] and mixed-precision arithmetic [37] to fully accelerate the AT with just 15 epochs. A line of research improves the performance and mitigates the catastrophic overfitting problem discovered in the Fast-AT, e.g., YOPO [63], GradAlign [2], GAT [45], Sub-AT [30], etc., but there are limited explorations on whether these recipes are compatible with the full ImageNet [12]. Although Fast-AT provides competitive PGD results, its resulting robustness on ResNet-50 is inferior to that of Standard-AT's as per the AA accuracy on the RobustBench leaderboard [7].…”

Section: Guideline 2 Followed Guideline 2 Followedmentioning

confidence: 99%

RobArch: Designing Robust Architectures against Adversarial Attacks

Peng¹,

Xu²,

Cornelius³

et al. 2023

Preprint

View full text Add to dashboard Cite

Adversarial Training is the most effective approach for improving the robustness of Deep Neural Networks (DNNs). However, compared to the large body of research in optimizing the adversarial training process, there are few investigations into how architecture components affect robustness, and they rarely constrain model capacity. Thus, it is unclear where robustness precisely comes from. In this work, we present the first large-scale systematic study on the robustness of DNN architecture components under fixed parameter budgets. Through our investigation, we distill 18 actionable robust network design guidelines that empower model developers to gain deep insights. We demonstrate these guidelines' effectiveness by introducing the novel Robust Architecture (RobArch) model that instantiates the guidelines to build a family of top-performing models across parameter capacities against strong adversarial attacks. RobArch achieves the new state-of-the-art AutoAttack accuracy on the RobustBench ImageNet leaderboard. The code is available at https://github.com/ShengYun-Peng/RobArch.

show abstract

“…Adversarial training [4], [25] is considered to be the most effective way to defend against adversarial attacks by augmenting training data with adversarial examples. Since the generation of AEs is time-consuming, many variants of AT try to improve training efficiency.…”

Section: B Efficient Adversarial Trainingmentioning

confidence: 99%

Investigating Catastrophic Overfitting in Fast Adversarial Training: A Self-fitting Perspective

He¹,

Li²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Although fast adversarial training provides an efficient approach for building robust networks, it may suffer from a serious problem known as catastrophic overfitting (CO), where the multi-step robust accuracy suddenly collapses to zero. In this paper, we for the first time decouple the FGSM examples into data-information and self-information, which reveals an interesting phenomenon called "self-fitting". Self-fitting, i.e., DNNs learn the self-information embedded in single-step perturbations, naturally leads to the occurrence of CO. When self-fitting occurs, the network experiences an obvious "channel differentiation" phenomenon that some convolution channels accounting for recognizing self-information become dominant, while others for data-information are suppressed. In this way, the network learns to only recognize images with sufficient self-information and loses generalization ability to other types of data. Based on self-fitting, we provide new insight into the existing methods to mitigate CO and extend CO to multi-step adversarial training. Our findings reveal a self-learning mechanism in adversarial training and open up new perspectives for suppressing different kinds of information to mitigate CO.Impact Statement-Fast adversarial training is an effective and efficient adversarial training method. However, it is prone to instability and can lead to catastrophic overfitting (CO). In this paper, we have revealed for the first time the existence of model self-information in adversarial examples, and argue that fitting self-information (self-fitting) is one of the factors that contribute to CO. Our findings can further aid in the understanding of CO in fast adversarial training and even multi-step adversarial training, inspiring the generation of more stable and efficient adversarial training algorithms. The discovery of self-fitting is not only for adversarial attack but also helpful other methods that involved network information, like curriculum learning, active learning, and self-supervised learning.

show abstract

Subspace Adversarial Training

Cited by 44 publications

References 9 publications

Generation and Countermeasures of Adversarial Examples on Vision: A Survey

Generation and Countermeasures of Adversarial Examples on Vision: A Survey

RobArch: Designing Robust Architectures against Adversarial Attacks

Investigating Catastrophic Overfitting in Fast Adversarial Training: A Self-fitting Perspective

Contact Info

Product

Resources

About