Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs

Benz, J. Philipp; Ham, Sunny; Zhang, Chaoning; Karjauv, Adil; Kweon, In So

doi:10.48550/arxiv.2110.02797

Cited by 16 publications

(21 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, a number of works compare the robustness of ViTs to ResNets. While there are mixed findings on adversarial robustness [4,46], there is agreement that ViTs have stronger out-of-distribution generalization, likely due to self attention [5,39]. In contrast, our work focuses on relative robustness to noise in foreground and background regions.…”

Section: Modelsmentioning

confidence: 89%

A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes

Moayeri¹,

Pope²,

Balaji³

et al. 2022

Preprint

View full text Add to dashboard Cite

While datasets with single-label supervision have propelled rapid advances in image classification, additional annotations are necessary in order to quantitatively assess how models make predictions. To this end, for a subset of ImageNet samples, we collect segmentation masks for the entire object and 18 informative attributes. We call this dataset RIVAL10 (RIch Visual Attributes with Localization), consisting of roughly 26k instances over 10 classes. Using RIVAL10, we evaluate the sensitivity of a broad set of models to noise corruptions in foregrounds, backgrounds and attributes. In our analysis, we consider diverse state-of-the-art architectures (ResNets, Transformers) and training procedures (CLIP, SimCLR, DeiT, Adversarial Training). We find that, somewhat surprisingly, in ResNets, adversarial training makes models more sensitive to the background compared to foreground than standard training. Similarly, contrastively-trained models also have lower relative foreground sensitivity in both transformers and ResNets. Lastly, we observe intriguing adaptive abilities of transformers to increase relative foreground sensitivity as corruption level increases. Using saliency methods, we automatically discover spurious features that drive the background sensitivity of models and assess alignment of saliency maps with foregrounds. Finally, we quantitatively study the attribution problem for neural features by comparing feature saliency with ground-truth localization of semantic attributes.

show abstract

Section: Modelsmentioning

confidence: 89%

A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes

Moayeri¹,

Pope²,

Balaji³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Although rich literature exists related to the robustness of CNNs in the medical imaging domain, to the best of our knowledge, no such study exists for ViTs, making it an exciting as well challenging direction to explore. Recently, few attempts have been made to evaluate the robustness of ViTs to adversarial attacks for natural images [407]- [416]. The main conclusions of these attempts, ignoring their nuance difference, can be summarized as ViTs are more robust to adversarial attacks than CNNs.…”

Section: Adversarial Robustnessmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as de facto operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of the applications of Transformers in medical imaging covering various aspects, ranging from recently proposed architectural designs to unsolved issues. Specifically, we survey the use of Transformers in medical image segmentation, detection, classification, reconstruction, synthesis, registration, clinical report generation, and other tasks. In particular, for each of these applications, we develop taxonomy, identify application-specific challenges as well as provide insights to solve them, and highlight recent trends. Further, we provide a critical discussion of the field's current state as a whole, including the identification of key challenges, open problems, and outlining promising future directions. We hope this survey will ignite further interest in the community and provide researchers with an up-to-date reference regarding applications of Transformer models in medical imaging. Finally, to cope with the rapid development in this field, we intend to regularly update the relevant latest papers and their open-source implementations at https://github.com/fahadshamshad/awesome-transformers-in-medical-imaging.

show abstract

“…In addition to competitive classification performance, two other recent works [113,110] explore the robustness of CNNs, Vision Transformer, and MLPs. CNNs have been widely known to be vulnerable to adversarial attacks [114,115], that is, small additive perturbations of the input cause the CNN to misclassify a sample, causing serious concerns for security-sensitive applications.…”

Section: Model Performancementioning

confidence: 99%

“…But how about the MLP-Mixer? Benz et al [110] compare ResNet, ViT and MLP-Mixer under white-box attack (Table 5), as well as both query-based and transfer-based black-box attacks. In all cases, ViT is the most robust architecture, MLP is the next, while CNN is the least robust.…”

Section: Model Performancementioning

confidence: 99%

See 1 more Smart Citation

Are we ready for a new paradigm shift? A Survey on Visual Deep MLP

Liu¹,

Li²,

Tao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Multilayer perceptron (MLP), as the first neural network structure to appear, was a big hit. But constrained by the hardware computing power and the size of the datasets, it once sank for tens of years. During this period, we have witnessed a paradigm shift from manual feature extraction to the CNN with local receptive field, and further to the Transformer with global receptive field based on selfattention mechanism. And this year (2021), with the introduction of MLP-Mixer, MLP has re-entered the limelight and has attracted extensive research from the computer vision community. Compare to the conventional MLP, it gets deeper but changes the input from full flattening to patch flattening. Given its high performance and less need for vision-specific inductive bias, the community can't help but wonder, Will deep MLP, the simplest structure with global receptive field but no attention, become a new computer vision paradigm? To answer this question, this survey aims to provide a comprehensive overview of the recent development of deep MLP models in vision. Specifically, we review these MLPs in detail, from the subtle sub-module design to the global network structure. We compare the receptive field, computational complexity, and other properties of different network designs in order to understand the development path of MLPs clearly. The investigation shows that MLPs' resolution-sensitivity and computational densities remain unresolved, and pure MLPs are gradually evolving towards CNN-like. We suggest that the current data volume and computational power are not ready to embrace pure MLPs, and artificial visual guidance remains important. Finally, we provide our viewpoint about open research directions and potential future works. We hope this effort will ignite further interest in the community and encourage better visual tailored design for the neural network in the future.

show abstract

Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs

Cited by 16 publications

References 52 publications

A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes

A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes

Transformers in Medical Imaging: A Survey

Are we ready for a new paradigm shift? A Survey on Visual Deep MLP

Contact Info

Product

Resources

About