On the Robustness of Vision Transformers to Adversarial Examples

Mahmood, Kaleel; Mahmood, Rigel; Dijk, Marten van

doi:10.48550/arxiv.2104.02610

Cited by 18 publications

(22 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Concurrent works on similar topics. Recently, there are a line of works [2,4,20,44,45,49,50,54,61,72] that have investigated transformers from the perspective of adversarial robustness. Specifically, [2,4,44,49,54,61] concurrently compare the robustness of transformers to CNNs and independently derive conclusions resembling each other.…”

Section: Related Workmentioning

confidence: 99%

Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs

Benz¹,

Ham²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Convolutional Neural Networks (CNNs) have become the de facto gold standard in computer vision applications in the past years. Recently, however, new model architectures have been proposed challenging the status quo. The Vision Transformer (ViT) relies solely on attention modules, while the MLP-Mixer architecture substitutes the selfattention modules with Multi-Layer Perceptrons (MLPs). Despite their great success, CNNs have been widely known to be vulnerable to adversarial attacks, causing serious concerns for security-sensitive applications. Thus, it is critical for the community to know whether the newly proposed ViT and MLP-Mixer are also vulnerable to adversarial attacks. To this end, we empirically evaluate their adversarial robustness under several adversarial attack setups and benchmark them against the widely used CNNs. Overall, we find that the two architectures, especially ViT, are more robust than their CNN models. Using a toy example, we also provide empirical evidence that the lower adversarial robustness of CNNs can be partially attributed to their shift-invariant property. Our frequency analysis suggests that the most robust ViT architectures tend to rely more on low-frequency features compared with CNNs. Additionally, we have an intriguing finding that MLP-Mixer is extremely vulnerable to universal adversarial perturbations. Code: https://github.com/phibenz/robustness_comparison_ vit_mlp-mixer_cnn. IntroductionConvolutional Neural Networks (CNNs) [37] have been the gold standard architecture in computer vision. In Natural Language Processing (NLP), however, attention-based transformers are the dominant go-to model architecture [13,55,56]. Various attempts have been made to apply such transformer architectures to computer vision tasks [8,10,53,58]. A

show abstract

Section: Related Workmentioning

confidence: 99%

Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs

Benz¹,

Ham²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Not only for tasks such as objection detection [27] and segmentation [27,69,70] Transformer-based achieves better performance than CNN-based, but also achieves state-of-the-art performance on denoising [71], deraining [71], super-resolution [71,72,73,74], etc. Furthermore, there is a line of works [75,76,77,78] showing that the Transformer-based architecture has better robustness than CNN-based. All the developments over the year indicate that the paradigm of computer vision is shifting from CNN to Transformer.…”

Section: Paradigm Shifts In the Last Decadementioning

confidence: 99%

“…CNNs have been widely known to be vulnerable to adversarial attacks [114,115], that is, small additive perturbations of the input cause the CNN to misclassify a sample, causing serious concerns for security-sensitive applications. And recently, there is a line of works [75,76,77,78] that have investigated Transformers from the perspective of adversarial robustness. Their main conclusions can be summarized as the Vision Transformers are more robust than CNNs.…”

Section: Model Performancementioning

confidence: 99%

Are we ready for a new paradigm shift? A Survey on Visual Deep MLP

Liu¹,

Li²,

Tao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Multilayer perceptron (MLP), as the first neural network structure to appear, was a big hit. But constrained by the hardware computing power and the size of the datasets, it once sank for tens of years. During this period, we have witnessed a paradigm shift from manual feature extraction to the CNN with local receptive field, and further to the Transformer with global receptive field based on selfattention mechanism. And this year (2021), with the introduction of MLP-Mixer, MLP has re-entered the limelight and has attracted extensive research from the computer vision community. Compare to the conventional MLP, it gets deeper but changes the input from full flattening to patch flattening. Given its high performance and less need for vision-specific inductive bias, the community can't help but wonder, Will deep MLP, the simplest structure with global receptive field but no attention, become a new computer vision paradigm? To answer this question, this survey aims to provide a comprehensive overview of the recent development of deep MLP models in vision. Specifically, we review these MLPs in detail, from the subtle sub-module design to the global network structure. We compare the receptive field, computational complexity, and other properties of different network designs in order to understand the development path of MLPs clearly. The investigation shows that MLPs' resolution-sensitivity and computational densities remain unresolved, and pure MLPs are gradually evolving towards CNN-like. We suggest that the current data volume and computational power are not ready to embrace pure MLPs, and artificial visual guidance remains important. Finally, we provide our viewpoint about open research directions and potential future works. We hope this effort will ignite further interest in the community and encourage better visual tailored design for the neural network in the future.

show abstract

“…The robustness of ViT have achieved great attention due to its great success in many vision tasks [2,4,5,20,23,26,28,29,31,35]. On the one hand, [5,31] show that vision transformers are more robust to natural corruptions [18] compared to CNNs.…”

Section: Robustness Of Vision Transformermentioning

confidence: 99%

Are Vision Transformers Robust to Patch Perturbations?

Gu¹,

Tresp²,

Qin³

2021

Preprint

View full text Add to dashboard Cite

The recent advances in Vision Transformer (ViT) have demonstrated its impressive performance in image classification, which makes it a promising alternative to Convolutional Neural Network (CNN). Unlike CNNs, ViT represents an input image as a sequence of image patches. The patch-wise input image representation makes the following question interesting: How does ViT perform when individual input image patches are perturbed with natural corruptions or adversarial perturbations, compared to CNNs? In this work, we study the robustness of vision transformers to patch-wise perturbations. Surprisingly, we find that vision transformers are more robust to naturally corrupted patches than CNNs, whereas they are more vulnerable to adversarial patches. Furthermore, we conduct extensive qualitative and quantitative experiments to understand the robustness to patch perturbations. We have revealed that ViT's stronger robustness to natural corrupted patches and higher vulnerability against adversarial patches are both caused by the attention mechanism. Specifically, the attention model can help improve the robustness of vision transformers by effectively ignoring natural corrupted patches. However, when vision transformers are attacked by an adversary, the attention mechanism can be easily fooled to focus more on the adversarially perturbed patches and cause a mistake.

show abstract

On the Robustness of Vision Transformers to Adversarial Examples

Cited by 18 publications

References 0 publications

Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs

Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs

Are we ready for a new paradigm shift? A Survey on Visual Deep MLP

Are Vision Transformers Robust to Patch Perturbations?

Contact Info

Product

Resources

About