Understanding Robustness of Transformers for Image Classification

Bhojanapalli, Srinadh; Chakrabarti, Ayan; Gläsner, Daniel; Li, Daliang; Unterthiner, Thomas; Veit, Andreas

doi:10.48550/arxiv.2103.14586

Cited by 30 publications

(69 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…tible noises to original data [1,22,32]. The characteristics of using non-overlapping patches in ViTs reduces the influence of adversarial examples with the same magnitude on the overall results [2].…”

Section: Boundary Heatmap Par Heatmapmentioning

confidence: 99%

“…In the image classification task, the decision-based attacks [4,10,34] start from a ran-dom noise with a large noise magnitude, randomly sample in the image input space, and gradually compress the adversarial noise under the premise of ensuring misclassification. The existing adversarial attacks against transformers are only white-box attacks [2,26] and transfer-based blackbox attacks [21,33]. Black-box attacks against ViTs without substitute model remains an open problem.…”

Section: Boundary Heatmap Par Heatmapmentioning

confidence: 99%

“…ViTs learn fewer low-level features and more transferable features than CNNs [26]. Meanwhile, ViTs split the image into multiple non-overlapping patches which reduce the impact of noises on one single patch to the final classification results [2]. Therefore, to find the initial adversarial examples that fool ViTs, the decision-based attacks have to add random noise with much larger magnitude than on the CNNs.…”

Section: Boundary Heatmap Par Heatmapmentioning

confidence: 99%

“…The ViTs not only achieve promising performance in multiple tasks [2], but also show stronger adversarial robustness [24]. Not only does fooling ViTs in the whitebox scenario require larger noise magnitude [26], but it is also difficult for existing transfer-based black-box attacks to transfer adversarial examples from CNNs to ViTs [33].…”

Section: Robustness Of Vision Transformermentioning

confidence: 99%

“…Vision transformers (ViTs) [11] not only achieve significant performance improvement in a wide range of computer vision tasks [5,17,29], but also show stronger robustness against adversarial examples generated by different attack methods [2,19,20] Boundary Noise = .…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removal

Shi¹,

Han²

2021

Preprint

View full text Add to dashboard Cite

Vision transformers (ViTs) have demonstrated impressive performance and stronger adversarial robustness compared to Deep Convolutional Neural Networks (CNNs). On the one hand, ViTs' focus on global interaction between individual patches reduces the local noise sensitivity of images. On the other hand, the existing decision-based attacks for CNNs ignore the difference in noise sensitivity between different regions of the image, which affects the efficiency of noise compression. Therefore, validating the black-box adversarial robustness of ViTs when the target model can only be queried still remains a challenging problem. In this paper, we propose a new decisionbased black-box attack against ViTs termed Patch-wise Adversarial Removal (PAR). PAR divides images into patches through a coarse-to-fine search process and compresses the noise on each patch separately. PAR records the noise magnitude and noise sensitivity of each patch and selects the patch with the highest query value for noise compression. In addition, PAR can be used as a noise initialization method for other decision-based attacks to improve the noise compression efficiency on both ViTs and CNNs without introducing additional calculations. Extensive experiments on ImageNet-21k, ILSVRC-2012, and Tiny-Imagenet datasets demonstrate that PAR achieves a much lower magnitude of perturbation on average with the same number of queries. Our code is available at https://github.com/shiyuchengTJU/PAR.

show abstract

Section: Boundary Heatmap Par Heatmapmentioning

confidence: 99%

Section: Boundary Heatmap Par Heatmapmentioning

confidence: 99%

Section: Boundary Heatmap Par Heatmapmentioning

confidence: 99%

Section: Robustness Of Vision Transformermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removal

Shi¹,

Han²

2021

Preprint

View full text Add to dashboard Cite

show abstract

ATTACK-COSM: attacking the camouflaged object segmentation model through digital world adversarial examples

Li,

Wang,

Zhang

et al. 2024

Complex Intell. Syst.

View full text Add to dashboard Cite

The camouflaged object segmentation model (COSM) has recently gained substantial attention due to its remarkable ability to detect camouflaged objects. Nevertheless, deep vision models are widely acknowledged to be susceptible to adversarial examples, which can mislead models, causing them to make incorrect predictions through imperceptible perturbations. The vulnerability to adversarial attacks raises significant concerns when deploying COSM in security-sensitive applications. Consequently, it is crucial to determine whether the foundational vision model COSM is also susceptible to such attacks. To our knowledge, our work represents the first exploration of strategies for targeting COSM with adversarial examples in the digital world. With the primary objective of reversing the predictions for both masked objects and backgrounds, we explore the adversarial robustness of COSM in full white-box and black-box settings. In addition to the primary objective of reversing the predictions for masked objects and backgrounds, our investigation reveals the potential to generate any desired mask through adversarial attacks. The experimental results indicate that COSM demonstrates weak robustness, rendering it vulnerable to adversarial example attacks. In the realm of COS, the projected gradient descent (PGD) attack method exhibits superior attack capabilities compared to the fast gradient sign (FGSM) method in both white-box and black-box settings. These findings reduce the security risks in the application of COSM and pave the way for multiple applications of COSM.

show abstract

Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers

Wang

Xie

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

117

View full text Add to dashboard Cite

Panoptic segmentation involves a combination of joint semantic segmentation and instance segmentation, where image contents are divided into two types: things and stuff. We present Panoptic SegFormer, a general framework for panoptic segmentation with transformers. It contains three innovative components: an efficient deeply-supervised mask decoder, a query decoupling strategy, and an improved post-processing method. We also use Deformable DETR to efficiently process multi-scale features, which is a fast and efficient version of DETR. Specifically, we supervise the attention modules in the mask decoder in a layer-wise manner. This deep supervision strategy lets the attention modules quickly focus on meaningful semantic regions. It improves performance and reduces the number of required training epochs by half compared to Deformable DETR. Our query decoupling strategy decouples the responsibilities of the query set and avoids mutual interference between things and stuff. In addition, our postprocessing strategy improves performance without additional costs by jointly considering classification and segmentation qualities to resolve conflicting mask overlaps. Our approach increases the accuracy 6.2% PQ over the baseline DETR model. Panoptic SegFormer achieves stateof-the-art results on COCO test-dev with 56.2% PQ. It also shows stronger zero-shot robustness over existing methods. The code is released at https://github.com/ zhiqi-li/Panoptic-SegFormer.

show abstract

Understanding Robustness of Transformers for Image Classification

Cited by 30 publications

References 34 publications

Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removal

Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removal

ATTACK-COSM: attacking the camouflaged object segmentation model through digital world adversarial examples

Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers

Contact Info

Product

Resources

About