The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Training

Líu, Hao; Jiang, Xinghua; Li, Xin; Guo, Antai; Jiang, Deqiang; Ren, Bo

doi:10.48550/arxiv.2204.08227

Cited by 3 publications

(4 citation statements)

References 43 publications

(87 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, as a representative of such methods, the reconstruction quality of MAE [27] is poor: fine details and textures are missing (Figure 2). A similar issue exists in many other MIM methods [11,36].…”

Section: Introductionmentioning

confidence: 68%

MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis

Li¹,

Chang²,

Mishra³

et al. 2022

Preprint

View full text Add to dashboard Cite

Generative modeling and representation learning are two key tasks in computer vision. However, these models are typically trained independently, which ignores the potential for each task to help the other, and leads to training and model maintenance overheads. In this work, we propose MAsked Generative Encoder (MAGE), the first framework to unify SOTA image generation and self-supervised representation learning. Our key insight is that using variable masking ratios in masked image modeling pre-training can allow generative training (very high masking ratio) and representation learning (lower masking ratio) under the same training framework. Inspired by previous generative models, MAGE uses semantic tokens learned by a vector-quantized GAN at inputs and outputs, combining this with masking. We can further improve the representation by adding a contrastive loss to the encoder output. We extensively evaluate the generation and representation learning capabilities of MAGE. On ImageNet-1K, a single MAGE ViT-L model obtains 9.10 FID in the task of classunconditional image generation and 78.9% top-1 accuracy for linear probing, achieving state-of-the-art performance in both image generation and representation learning. Code is available at https://github.com/LTH14/mage.

show abstract

Section: Introductionmentioning

confidence: 68%

MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis

Li¹,

Chang²,

Mishra³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…With the development of vision transformers [18,23,39,50,55], Masked Image Modeling (MIM) gradually replaces the dominant position of contrastive learning [10,25,54] in visual self-supervised representation learning due to its superior fine-tuning performance in various visual downstream tasks. Many target signals have been designed for the mask-prediction pretext task in MIM, such as normalized pixels [24,60], discrete tokens [2,17], HOG feature [57], deep features [1,67] or frequencies [38,59]. However, they are all only applied as single-scale supervisions for reconstruction.…”

Section: Related Workmentioning

confidence: 99%

“…MIM learns semantic representations by first masking some parts of the input and then predicting their signals based on the unmasked parts, e.g., normalized pixels [24,60], discrete tokens [2,17], HOG fea-* Corresponding author. ture [57], deep features [1,67] or frequencies [38,59].…”

Section: Introductionmentioning

confidence: 99%

Masked Image Modeling with Local Multi-Scale Reconstruction

Wang¹,

Wang²,

Guo³

et al. 2023

Preprint

View full text Add to dashboard Cite

Masked Image Modeling (MIM) achieves outstanding success in self-supervised representation learning. Unfortunately, MIM models typically have huge computational burden and slow learning process, which is an inevitable obstacle for their industrial applications. Although the lower layers play the key role in MIM, existing MIM models conduct reconstruction task only at the top layer of encoder. The lower layers are not explicitly guided and the interaction among their patches is only used for calculating new activations. Considering the reconstruction task requires non-trivial inter-patch interactions to reason target signals, we apply it to multiple local layers including lower and upper layers. Further, since the multiple layers expect to learn the information of different scales, we design local multiscale reconstruction, where the lower and upper layers reconstruct fine-scale and coarse-scale supervision signals respectively. This design not only accelerates the representation learning process by explicitly guiding multiple layers, but also facilitates multi-scale semantical understanding to the input. Extensive experiments show that with significantly less pre-training burden, our model achieves comparable or better performance on classification, detection and segmentation tasks than existing MIM models. Code is available with both MindSpore and PyTorch.

show abstract

“…Besides, Fang et al [22] employ an auxiliary generator to corrupt the input images. As to what to predict, beyond default raw pixels [26,79], several other reconstruction targets are proposed, e.g., hand-crafted or deep features [65], low or high frequencies [38,74], and discrete tokens [2]. Correlational modeling is the crucial process in visual tracking [81], aiming to predict a dense set of matching confidence for a target object.…”

Section: Related Workmentioning

confidence: 99%