SLIP: Self-supervision Meets Language-Image Pre-training

Mu, Norman; Kirillov, Alexander; Wagner, David; Xie, Sihong

doi:10.1007/978-3-031-19809-0_30

Cited by 128 publications

(65 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The core technique of CLIP is aligning both vision and language modalities in a joint embedding space by global representation contrastive. Followup works further improve CLIP from the vision-only [43,67] or vision-language [38,64,68,69] side. In this paper, we bring natural language supervision together with masked image modeling for better visual pre-training on these two paradigms.…”

Section: Related Workmentioning

confidence: 99%

“…We evaluate the zero-shot classification over 21 benchmarks including ImageNet-1K [15]. Detail of each datasets are listed in the appendix and the evaluate recipes (e.g., prompt engineering) strictly follow [43]. Results are shown in Table 6.…”

Section: Zero-shot Transfermentioning

confidence: 99%

“…RILS also demonstrates superior performance on zero-shot image classification and image-text retrieval. On ImageNet-1K benchmark, RILS obtains 45.0% zero-shot classification accuracy, +4.7%/+3.4% higher than CLIP [47]/SLIP [43] using the same training data. Compelling results of RILS imply the promising out-of-the-box capacity in the unification of MIM and language supervision.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

RILS: Masked Visual Reconstruction in Language Semantic Space

Yang¹,

Ge²,

Yi³

et al. 2023

Preprint

View full text Add to dashboard Cite

Both masked image modeling (MIM) and natural language supervision have facilitated the progress of transferable visual pre-training. In this work, we seek the synergy between two paradigms and study the emerging properties when MIM meets natural language supervision. To this end, we present a novel masked visual Reconstruction In Language semantic Space (RILS) pre-training framework, in which sentence representations, encoded by the text encoder, serve as prototypes to transform the vision-only signals into patch-sentence probabilities as semantically meaningful MIM reconstruction targets. The vision models can therefore capture useful components with structured information by predicting proper semantic of masked tokens. Better visual representations could, in turn, improve the text encoder via the image-text alignment objective, which is essential for the effective MIM target transformation. Extensive experimental results demonstrate that our method not only enjoys the best of previous MIM and CLIP but also achieves further improvements on various tasks due to their mutual benefits. RILS exhibits advanced transferability on downstream classification, detection, and segmentation, especially for low-shot regimes. Code will be made available at https://github.com/hustvl/RILS.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Zero-shot Transfermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

RILS: Masked Visual Reconstruction in Language Semantic Space

Yang¹,

Ge²,

Yi³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…It is relatively easy to collect a large-scale dataset of image-text paired data from the web [34,35]; Many works studied to enhance the learned visual representations of the model with web image-text datasets [26,27,30,32,54,55]. CLIP [32] tackles this challenge by performing contrastive learning over the paired image and text data.…”

Section: Vision Learners With Web Image-text Pairsmentioning

confidence: 99%

“…Differing from the above single-modal SSL methods, pioneering multi-modal SSL methods (CLIP [32] and ALIGN [24]) have shown great scalability, i.e., easily collected large-scale web image-text pairs [34,35,38] can contribute to great transfer learning performances. The following works also proved that the textual information in web image-text pairs could greatly benefit various downstream tasks [26,27,30,54]. In a fair condition, with the same web image-text dataset and training schedule, we first conduct a benchmark study of single-modal SSL methods (with only images) and multi-modal SSL methods (with imagetext pairs).…”

Section: Introductionmentioning

confidence: 99%

Vision Learners Meet Web Image-Text Pairs

Zhao¹,

Quan²,

Wu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Most recent self-supervised learning (SSL) methods are pre-trained on the well-curated ImageNet-1K dataset. In this work, we consider SSL pre-training on noisy web image-text paired data due to the excellent scalability of web data. First, we conduct a benchmark study of representative SSL pre-training methods on large-scale web data in a fair condition. Methods include single-modal ones such as MAE and multi-modal ones such as CLIP. We observe that multi-modal methods cannot outperform singlemodal ones on vision transfer learning tasks. We derive an information-theoretical view to explain the benchmarking results, which provides insights into designing novel vision learners. Inspired by the above explorations, we present a visual representation pre-training method, MUlti-modal Generator (MUG), for scalable web image-text data. MUG achieves state-of-the-art transferring performances on a variety of tasks and shows promising scaling behavior. Models and codes will be made public. 1

show abstract

Correction to: Processing-in-Memory for AI

Kim

2022

Processing-in-Memory for AI

View full text Add to dashboard Cite

show abstract

SLIP: Self-supervision Meets Language-Image Pre-training

Cited by 128 publications

References 23 publications

RILS: Masked Visual Reconstruction in Language Semantic Space

RILS: Masked Visual Reconstruction in Language Semantic Space

Vision Learners Meet Web Image-Text Pairs

Correction to: Processing-in-Memory for AI

Contact Info

Product

Resources

About