Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation

Cheng, Ruizhe; Wu, Bichen; Zhang, Peizhao; Vajda, Péter

doi:10.1109/cvprw53098.2021.00348

Cited by 25 publications

(15 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, benefiting from contrastive training [65] and the scalability of modern backbones, CLIP [43] and ALIGN [30] learn strong visual representations on large-scale image-text datasets, advancing the transfer performance on the downstream vision tasks. Later works improve CLIP by introducing more auxiliary loss functions to assist the image-text contrastive loss, such as image self-supervision loss [40,34], self-distillation loss [12], and token-wise max similarity [62].…”

Section: Related Workmentioning

confidence: 99%

MILAN: Masked Image Pretraining on Language Assisted Representation

Hou¹,

Sun²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Self-attention based transformer models have been dominating many computer vision tasks in the past few years. Their superb model qualities heavily depend on the excessively large labeled image datasets. In order to reduce the reliance on large labeled datasets, reconstruction based masked autoencoders are gaining popularity, which learn high quality transferable representations from unlabeled images. For the same purpose, recent weakly supervised image pretraining methods explore language supervision from text captions accompanying the images. In this work, we propose masked image pretraining on language assisted representation, dubbed as MILAN. Instead of predicting raw pixels or low level features, our pretraining objective is to reconstruct the image features with substantial semantic signals that are obtained using caption supervision. Moreover, to accommodate our reconstruction target, we propose a more efficient prompting decoder architecture and a semantic aware mask sampling mechanism, which further advance the transfer performance of the pretrained model. Experimental results demonstrate that MILAN delivers higher accuracy than the previous works. When the masked autoencoder is pretrained and finetuned on ImageNet-1K dataset with an input resolution of 224×224, MILAN achieves a top-1 accuracy of 85.4% on ViT-B/16, surpassing previous state-of-the-arts by 1%. In the downstream semantic segmentation task, MILAN achieves 52.7 mIoU using ViT-B/16 backbone on ADE20K dataset, outperforming previous masked pretraining results by 4 points 1 .

show abstract

Section: Related Workmentioning

confidence: 99%

MILAN: Masked Image Pretraining on Language Assisted Representation

Hou¹,

Sun²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…ALIGN [18] further scales-up the pre-traininig dataset to noisier 1B data points. [7] proposes a distillation based loss to better handle the noise in the dataset. Efficient-CLIP [44] is a concurrent work that uses text-only data to perform unimodal MLM task.…”

Section: Related Workmentioning

confidence: 99%

A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision

Tejankar¹,

Sanjabi²,

Wu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Using natural language as a supervision for training visual recognition models holds great promise. Recent works have shown that if such supervision is used in the form of alignment between images and captions in large training datasets, then the resulting aligned models perform well on zero-shot classification as downstream tasks 2 . In this paper, we focus on teasing out what parts of the language supervision are essential for training zero-shot image classification models. Through extensive and careful experiments, we show that: 1) A simple Bag-of-Words (BoW) caption could be used as a replacement for most of the image captions in the dataset. Surprisingly, we observe that this approach improves the zero-shot classification performance when combined with word balancing. 2) Using a BoW pretrained model, we can obtain more training data by generating pseudo-BoW captions on images that do not have a caption. Models trained on images with real and pseudo-BoW captions achieve stronger zero-shot performance. On ImageNet-1k zero-shot evaluation, our best model, that uses only 3M image-caption pairs, performs on-par with a CLIP model trained on 15M image-caption pairs (31.5% vs 31.3%).

show abstract

“…Knowledge Distillation: In conventional knowledge distillation [12,17,24,26,30,33,37,44,47,50], a larger model serves as the teacher to a smaller student model. In self-distillation scenarios [11,14,28], the size of the teacher model is typically comparable to student model. The goal is usually to obtain a computationally lighter and more efficient framework, but still maintain similar or even higher accuracy.…”

Section: Related Workmentioning

confidence: 99%

CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks

Wang¹,

Codella²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Contrastive language-image pretraining (CLIP) links vision and language modalities into a unified embedding space, yielding tremendous potential for vision-language (VL) tasks. While early concurrent works have begun to study this potential on a subset of tasks, important questions remain: 1) What is the benefit of CLIP on unstudied VL tasks? 2) Does CLIP provide benefit in low-shot or domain shifted scenarios? 3) Can CLIP improve existing approaches without impacting inference or pretraining complexity? In this work, we seek to answer these questions through two key contributions. First, we introduce an evaluation protocol that includes Visual Commonsense Reasoning (VCR), Visual Entailment (SNLI-VE), and Visual Question Answering (VQA), across a variety of data availability constraints and conditions of domain shift. Second, we propose an approach, named CLIP Targeted Distillation (CLIP-TD), to intelligently distill knowledge from CLIP into existing architectures using a dynamically weighted objective applied to adaptively selected tokens per instance. Experiments demonstrate that our proposed CLIP-TD leads to exceptional gains in the low-shot (up to 51.9%) and domain-shifted (up to 71.3%) conditions of VCR, while simultaneously improving performance under standard fullysupervised conditions (up to 2%), achieving state-of-art performance on VCR compared to other single models that are pretrained with image-text data only. On SNLI-VE, CLIP-TD produces significant gains in low-shot conditions (up to 6.6%) as well as fully supervised (up to 3%). On VQA, CLIP-TD provides improvement in low-shot (up to 9%), and in fully-supervised (up to 1.3%). Finally, CLIP-TD outperforms concurrent works utilizing CLIP for finetuning, as well as baseline naive distillation approaches. Code will be made available.

show abstract

Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation

Cited by 25 publications

References 20 publications

MILAN: Masked Image Pretraining on Language Assisted Representation

MILAN: Masked Image Pretraining on Language Assisted Representation

A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision

CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks

Contact Info

Product

Resources

About