Siamese Image Modeling for Self-Supervised Vision Representation Learning

Tao, Chenxin; Zhu, Xizhou; Huang, Gao; Qiao, Yu; Wang, Xiaogang; Dai, Jifeng

doi:10.48550/arxiv.2206.01204

Cited by 4 publications

(11 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A study of the Variance-Invariance-Covariance pattern for the output of each block is conducted in order to better understand the Layer Grafted Pretraining. As illustrated in Figure 6, we find that the VIC pattern of Layer Grafted Pre-training tends 83.2 SIM (Tao et al, 2022) 83.8 ConMIM (Yi et al, 2022) 83.7 Layer Grafted Pre-training (Ours) 83.9…”

Section: More Ablationsmentioning

confidence: 86%

“…We start by verifying the effectiveness of the proposed Layer Grafted Pre-training by comparing it with state-of-the-art methods. As shown in (Zhou et al, 2021) 76.0 --SIM (Tao et al, 2022) 76.4 65.3 -C-MAE 73.9 65.3 77.3 MimCo 70 shows better intra-variance: the red (•), green (•) and yellow (•) points of Moco V3 collapse to a smaller region than the proposed Layer Grafted Pre-training.…”

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 94%

See 1 more Smart Citation

Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations

Jiang¹,

Chen²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recently, both Contrastive Learning (CL) and Mask Image Modeling (MIM) demonstrate that self-supervision is powerful to learn good representations. However, naively combining them is far from success. In this paper, we start by making the empirical observation that a naive joint optimization of CL and MIM losses leads to conflicting gradient directions -more severe as the layers go deeper. This motivates us to shift the paradigm from combining loss at the end, to choosing the proper learning method per network layer. Inspired by experimental observations, we find that MIM and CL are suitable to lower and higher layers, respectively. We hence propose to combine them in a surprisingly simple, "sequential cascade" fashion: early layers are first trained under one MIM loss, on top of which latter layers continue to be trained under another CL loss. The proposed Layer Grafted Pre-training learns good visual representations that demonstrate superior label efficiency in downstream applications, in particular yielding strong few-shot performance besides linear evaluation. For instance, on ImageNet-1k, Layer Grafted Pre-training yields 65.5% Top-1 accuracy in terms of 1% few-shot learning with ViT-B/16, which improves MIM and CL baselines by 14.4% and 2.1% with no bells and whistles. The code is available at https://github. com/VITA-Group/layerGraftedPretraining_ICLR23.git.

show abstract

Section: More Ablationsmentioning

confidence: 86%

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 94%

Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations

Jiang¹,

Chen²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Recent work attempted to combine the power of contrastive learning and masked image modelling. SIM (Tao et al, 2022) incorporated masking as part of the data augmentation operations into the contrastive learning framework. iBOT (Zhou et al, 2021) adopted a siamese network structure as contrastive learning and minimize the distance between the masked branch and the unmasked branch.…”

Section: Related Workmentioning

confidence: 99%

“…We fine-tuned the pre-trained models (already finetuned on ImageNet-1K) for 160K steps on ADE20K. The pre-trained model is delivered using ViT-Large MAE as the teacher and (Tao et al, 2022).…”

Section: Transfer Learning On Downstream Taskmentioning

confidence: 99%

MOMA:Distill from Self-Supervised Teachers

Yao¹,

Desai²,

Palaniswami³

2023

Preprint

View full text Add to dashboard Cite

Contrastive Learning and Masked Image Modelling have demonstrated exceptional performance on self-supervised representation learning, where Momentum Contrast (i.e., MoCo) and Masked AutoEncoder (i.e., MAE) are the stateof-the-art, respectively. In this work, we propose MOMA to distill from pre-trained MOCo and MAE in a self-supervised manner to collaborate the knowledge from both paradigms. During the distillation, the teacher and the student are fed with original inputs and masked inputs, respectively. The learning is enabled by aligning the normalized representations from the teacher and the projected representations from the student. This simple design leads to efficient computation with extremely high mask ratio and dramatically reduced training epochs, and does not require extra considerations on the distillation target. The experiments show MOMA delivers compact student models with comparable performance to existing state-of-the-art methods, combining the power of both self-supervised learning paradigms. It presents competitive results against different benchmarks in computer vision. We hope our method provides an insight on transferring and adapting the knowledge from large-scale pre-trained models in a computationally efficient way.Recent studies (Chung et al., 2021) (Mishra et al., 2022 attempt to combine the power of contrastive learning and masked modelling, yielding promising results. They suggest that both paradigms are complementary with each other and can deliver stronger representations when they are combined into a unified framework. Furthermore, integrating two paradigms into one framework introduces higher computational cost, which requires extensive resources (e.g., hundreds of GPU hours, enormous memory capacity, and excessive storage requirements). It is also not energy-efficient to training different frameworks from the scratch as they

show abstract

“…For instance, propose a combination of contrastive and masked reconstruction objectives using one masked view, and one full (unmasked) view. Other recent works (Tao et al, 2022;Assran et al, 2022) use similar asymmetric designs. The key distinction between CAN and concurrent work is that we strike a different balance between simplicity, efficiency, and performance: we focus on developing a simple, efficient and symmetric method: we use two masked views and no momentum encoder.…”

Section: Concurrent Workmentioning

confidence: 99%

A simple, efficient and scalable contrastive masked autoencoder for learning visual representations

Mishra¹,

Robinson²,

Chang³

et al. 2022

Preprint

View full text Add to dashboard Cite

We introduce CAN, a simple, efficient and scalable method for self-supervised learning of visual representations. Our framework is a minimal and conceptually clean synthesis of (C) contrastive learning, (A) masked autoencoders, and (N) the noise prediction approach used in diffusion models. The learning mechanisms are complementary to one another: contrastive learning shapes the embedding space across a batch of image samples; masked autoencoders focus on reconstruction of the low-frequency spatial correlations in a single image sample; and noise prediction encourages the reconstruction of the high-frequency components of an image. The combined approach results in a robust, scalable and simple-to-implement algorithm. The training process is symmetric, with 50% of patches in both views being masked at random, yielding a considerable efficiency improvement over prior contrastive learning methods. Extensive empirical studies demonstrate that CAN achieves strong downstream performance under both linear and finetuning evaluations on transfer learning and robustness tasks. CAN outperforms MAE and SimCLR when pre-training on ImageNet, but is especially useful for pre-training on larger uncurated datasets such as JFT-300M: for linear probe on ImageNet, CAN achieves 75.4% compared to 73.4% for SimCLR and 64.1% for MAE. The finetuned performance on ImageNet of our ViT-L model is 86.1%, compared to 85.5% for SimCLR, and 85.4% for MAE. The overall FLOPs load of SimCLR is 70% higher than CAN for ViT-L models 1 . * Equal contribution, order chosen randomly. Work done during an internship at Google. 1 Code will be released.

show abstract

Siamese Image Modeling for Self-Supervised Vision Representation Learning

Cited by 4 publications

References 32 publications

Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations

Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations

MOMA:Distill from Self-Supervised Teachers

A simple, efficient and scalable contrastive masked autoencoder for learning visual representations

Contact Info

Product

Resources

About