VirTex: Learning Visual Representations from Textual Annotations

Desai, Karan; Johnson, Justin C.

doi:10.48550/arxiv.2006.06666

Cited by 29 publications

(50 citation statements)

References 79 publications

(157 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[28] shows that pretraining by predicting hashtags on Instagram improves performance on ImageNet by over 5%. [8,35,44] all demonstrate the effectiveness of transformer-based language modeling in learning image representation from text. CLIP [32] and ALIGN [21] apply natural language supervision to the domain of ZSL.…”

Section: Introductionmentioning

confidence: 88%

Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation

Cheng¹,

Wu²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Traditional computer vision models are trained to predict a fixed set of predefined categories. Recently, natural language has been shown to be a broader and richer source of supervision that provides finer descriptions to visual concepts than supervised "gold" labels. Previous works, such as CLIP, use a simple pretraining task of predicting the pairings between images and text captions. CLIP, however, is data hungry and requires more than 400M image text pairs for training. We propose a data-efficient contrastive distillation method that uses soft labels to learn from noisy image-text pairs. Our model transfers knowledge from pretrained image and sentence encoders and achieves strong performance with only 3M image text pairs, 133x smaller than CLIP. Our method exceeds the previous SoTA of general zero-shot learning on ImageNet 21k+1k by 73% relatively with a ResNet50 image encoder and DeCLUTR text encoder. We also beat CLIP by 10.5% relatively on zeroshot evaluation on Google Open Images (19,958 classes).

show abstract

Section: Introductionmentioning

confidence: 88%

Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation

Cheng¹,

Wu²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition, (Joulin* et al, 2016;Li et al, 2017;Desai & Johnson, 2020;Sariyildiz et al, 2020) demonstrate that good visual representations can be learned by predicting image captions. To scale up vision-language joint training, CLIP (Radford et al, 2021) and ALIGN both collect their own image-text datasets with 400M and 1B image-caption pairs.…”

Section: Related Workmentioning

confidence: 99%

“…Recently, natural language has been used as a powerful source of supervision for visual representation learning. (Desai & Johnson, 2020;Sariyildiz et al, 2020; demonstrate the effectiveness of pretraining on image-text data. Among them, CLIP (Radford et al, 2021) applies natural language supervision to zero-shot image recognition.…”

Section: Introductionmentioning

confidence: 99%

Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation

Wu¹,

Cheng²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Traditional computer vision models are trained to predict a fixed set of predefined categories. Recently, natural language has been shown to be a broader and richer source of supervision that provides finer descriptions to visual concepts than supervised "gold" labels. Previous works, such as CLIP, use InfoNCE loss to train a model to predict the pairing between images and text captions. CLIP, however, is data hungry and requires more than 400M image-text pairs for training. The inefficiency can be partially attributed to the fact that the image-text pairs are noisy. To address this, we propose OTTER (Optimal TransporT distillation for Efficient zero-shot Recognition), which uses online entropic optimal transport to find a soft image-text match as labels for contrastive learning. Based on pretrained image and text encoders, models trained with OTTER achieve strong performance with only 3M image text pairs. Compared with InfoNCE loss, label smoothing, and knowledge distillation, OTTER consistently outperforms these baselines in zero-shot evaluation on Google Open Images (19,958 classes) and multi-labeled ImageNet 10K (10032 classes) from Tencent ML-Images. Over 42 evaluations on 7 different dataset/architecture settings x 6 metrics, OTTER outperforms (32) or ties (2) all baselines in 34 of them. Our source code is open sourced at https: //github.com/facebookresearch/OTTER.

show abstract

“…There is an alternative approach that directly computes the similarity score without having modality-wise representation. A typical example is the cross-modal attention models [11,22,30,32] (details in Sec. 4).…”

Section: Problem Setup and Backgroundmentioning

confidence: 99%

“…The main benefit of the former is the computational efficiency, scalable to billions of instances at training/test time, thanks to the efficient dotproduct. The latter directly computes the similarity score without having modality-wise representation [11,22,30,32] using the transformer-like attentive neural networks which aim to capture interactions between local features in the instances from different modalities. Although they can capture cross-modal interactions between local features of data instances from different modalities, they are computationally demanding and very slow due to the quadratic complexity in the number of local features.…”

Section: Related Workmentioning

confidence: 99%

SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval

Kim¹

2021

Preprint

View full text Add to dashboard Cite

We tackle the cross-modal retrieval problem, where the training is only supervised by the relevant multi-modal pairs in the data. The contrastive learning is the most popular approach for this task. However, its sampling complexity for learning is quadratic in the number of training data points. Moreover, it makes potentially wrong assumption that the instances in different pairs are automatically irrelevant. To address these issues, we propose a novel loss function that is based on self-labeling of the unknown classes. Specifically, we aim to predict class labels of the data instances in each modality, and assign those labels to the corresponding instances in the other modality (i.e., swapping the pseudo labels). With these swapped labels, we learn the data embedding for each modality using the supervised cross-entropy loss, hence leading to linear sampling complexity. We also maintain the queues for storing the embeddings of the latest batches, for which clustering assignment and embedding learning are done at the same time in an online fashion. This removes computational overhead of injecting intermittent epochs of entire training data sweep for offline clustering. We tested our approach on several realworld cross-modal retrieval problems, including text-based video retrieval, sketch-based image retrieval, and imagetext retrieval, and for all these tasks our method achieves significant performance improvement over the contrastive learning.

show abstract

VirTex: Learning Visual Representations from Textual Annotations

Cited by 29 publications

References 79 publications

Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation

Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation

Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation

SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval

Contact Info

Product

Resources

About