Masked Autoencoders Are Scalable Vision Learners

He, Kai; Chen, Xinlei; Xie, Sihong; Li, Yanghao; Dollár, Piotr; Girshick, Ross

doi:10.1109/cvpr52688.2022.01553

Cited by 3,566 publications

(1,839 citation statements)

References 25 publications

Supporting

Mentioning

1,809

Contrasting

Unclassified

Order By: Relevance

“…Compared with plain ViT, AugReg-ViT and ViT with discrete representation called DrViT [41], DAT with AugReg achieves better performance. The best result is from ViT-Huge pretrained by MAE [19] and finetuned by DAT, which suggests DAT is also effective in downstream fine-tuning tasks.…”

Section: Resultsmentioning

confidence: 94%

See 1 more Smart Citation

Enhance the Visual Representation via Discrete Adversarial Training

Mao¹,

Chen²,

Duan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Adversarial Training (AT), which is commonly accepted as one of the most effective approaches defending against adversarial examples, can largely harm the standard performance, thus has limited usefulness on industrial-scale production and applications. Surprisingly, this phenomenon is totally opposite in Natural Language Processing (NLP) task, where AT can even benefit for generalization. We notice the merit of AT in NLP tasks could derive from the discrete and symbolic input space. For borrowing the advantage from NLP-style AT, we propose Discrete Adversarial Training (DAT). DAT leverages VQGAN to reform the image data to discrete text-like inputs, i.e. visual words. Then it minimizes the maximal risk on such discrete images with symbolic adversarial perturbations. We further give an explanation from the perspective of distribution to demonstrate the effectiveness of DAT. As a plug-and-play technique for enhancing the visual representation, DAT achieves significant improvement on multiple tasks including image classification, object detection and self-supervised learning. Especially, the model pre-trained with Masked Auto-Encoding (MAE) and fine-tuned by our DAT without extra data can get 31.40 mCE on ImageNet-C and 32.77% top-1 accuracy on Stylized-ImageNet, building the new state-of-the-art. The code will be available at https://github.com/alibaba/easyrobust.Preprint. Under review.

show abstract

Section: Resultsmentioning

confidence: 94%

“…For ViTs, we adopt ViT-B/16 as baseline models, which is trained by the recipes in AugReg [48]. Besides, we use DAT to conduct supervised finetuning on downstream ImageNet classification task based on a self-supervised ViT-Huge pretrained by MAE [19]. By default, we refer ViT to ViT-B/16 in all tables and figures.…”

Section: Image Classificationmentioning

confidence: 99%

Enhance the Visual Representation via Discrete Adversarial Training

Mao¹,

Chen²,

Duan³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In image inpainting, Deng et al proposed Contextual Transformer Network for improving the continuity of context. [16] proposed an decoder-encoder based transformer which name masked autoencoder. What's more, [9] develops an inpainting transformer for completing large missing regions, which is the state-of-the-art method up to now.…”

Section: Image Completionmentioning

confidence: 99%

“…We designed a transformer based neural network to achieve phase completion for fringe projection profiler. Our network is inspired by [16] and [9], which is the simplification of [9] due to memory cost and forward speed considerations. The model has three parts: a convolution-based encoder, a transformer-based module with contextual attention, and a feedforward reconstruction module.…”

Section: Completion Modelmentioning

confidence: 99%

Phase Completion for Fringe Projection Profiler Based on Neural Networks

2022

ACSS

View full text Add to dashboard Cite

Fringe projection profiler (FPP) measures the geometry of the target surface by projecting the pre-modulated stripe map onto the surface, and then capture the phase map with a camera. However, the inaccurate exposure or the characteristics of the surface reflectance may influence the imaging quality of the phase map, leaving some over-exposure and under-exposure regions. Addressing to this problem, this paper propose to apply a neural network to complete the phase map. Firstly, we propose a synthetic dataset to simulate the phase map of the inaccurate exposure regions, based on a physical rendering model. After that, we implement a transformer neural network to complete the missing phase information. Experiments show that the proposed neural network can complete the missing information from its neighbouring information, and provide precise completion results.

show abstract

“…But, the training setup e.g., batch size in original papers is not always affordable for research institutions. More recently, masked image modeling (MIM) methods represented by MAE [11] have been proved to learn rich visual representations and significantly improve the performance on downstream tasks [18]. After randomly adding masks to input images, a pixellevel regression target is set as a pretext task.…”

Section: Introductionmentioning

confidence: 99%

S$$^3$$R: Self-supervised Spectral Regression for Hyperspectral Histopathology Image Classification

Xie

Wang

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Benefited from the rich and detailed spectral information in hyperspectral images (HSI), HSI offers great potential for a wide variety of medical applications such as computational pathology. But, the lack of adequate annotated data and the high spatiospectral dimensions of HSIs usually make classification networks prone to overfit. Thus, learning a general representation which can be transferred to the downstream tasks is imperative. To our knowledge, no appropriate self-supervised pre-training method has been designed for histopathology HSIs. In this paper, we introduce an efficient and effective Self-supervised Spectral Regression (S 3 R) method, which exploits the low rank characteristic in the spectral domain of HSI. More concretely, we propose to learn a set of linear coefficients that can be used to represent one band by the remaining bands via masking out these bands. Then, the band is restored by using the learned coefficients to reweight the remaining bands. Two pre-text tasks are designed: (1) S 3 R-CR, which regresses the linear coefficients, so that the pre-trained model understands the inherent structures of HSIs and the pathological characteristics of different morphologies; (2) S 3 R-BR, which regresses the missing band, making the model to learn the holistic semantics of HSIs. Compared to prior arts i.e., contrastive learning methods, which focuses on natural images, S 3 R converges at least 3 times faster, and achieves significant improvements up to 14% in accuracy when transferring to HSI classification tasks.

show abstract

Masked Autoencoders Are Scalable Vision Learners

Cited by 3,566 publications

References 25 publications

Enhance the Visual Representation via Discrete Adversarial Training

Enhance the Visual Representation via Discrete Adversarial Training

Phase Completion for Fringe Projection Profiler Based on Neural Networks

S$$^3$$R: Self-supervised Spectral Regression for Hyperspectral Histopathology Image Classification

Contact Info

Product

Resources

About