CyCLIP: Cyclic Contrastive Language-Image Pretraining

Goel, Shashank; Bansal, Hritik; Bhatia, Sumit; Rossi, Ryan A.; Vinay, V.; Grover, Aditya

doi:10.48550/arxiv.2205.14459

Cited by 5 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PyramidCLIP (Gao et al 2022) andFILIP (Yao et al 2021) introduce finer-grained and more interactions between two modalities, seeking for more accurate cross-modal alignment. CyCLIP (Goel et al 2022) points out the importance of geometric consistency in the learned representation space between two modalities, and proposes geometrically consistency constraints. Different from dual-stream ones, single-stream models, such as Visual-BERT (Li et al 2019) and OSCAR (Li et al 2020), fuse the image and text features with a unified model to achieve deeper interaction.…”

Section: Related Work Vision Language Pre-trainingmentioning

confidence: 99%

SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger

Gao,

Liu,

et al. 2024

AAAI

View full text Add to dashboard Cite

During the preceding biennium, vision-language pre-training has achieved noteworthy success on several downstream tasks. Nevertheless, acquiring high-quality image-text pairs, where the pairs are entirely exclusive of each other, remains a challenging task, and noise exists in the commonly used datasets. To address this issue, we propose SoftCLIP, a novel approach that relaxes the strict one-to-one constraint and achieves a soft cross-modal alignment by introducing a softened target, which is generated from the fine-grained intra-modal self-similarity. The intra-modal guidance is indicative to enable two pairs have some local similarities and model many-to-many relationships between the two modalities. Besides, since the positive still dominates in the softened target distribution, we disentangle the negatives in the distribution to further boost the relation alignment with the negatives in the cross-modal learning. Extensive experiments demonstrate the effectiveness of SoftCLIP. In particular, on ImageNet zero-shot classification task, using CC3M/CC12M as pre-training dataset, SoftCLIP brings a top-1 accuracy improvement of 6.8%/7.2% over the CLIP baseline.

show abstract

Section: Related Work Vision Language Pre-trainingmentioning

confidence: 99%

SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger

Gao,

Liu,

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Recent advancements in contrastive learning have enabled CLIP [46] to perform multimodal learning with 400M noisy data crawled from the web. CLIP has been extended for high efficiency model training and cycle consistency through various methods, such as AL-BEF [24] and Cyclip [15]. BLIP [23] includes text-to-image generation as an auxiliary task, which results in better performance by utilizing synthetic data as a bonus.…”

Section: Related Workmentioning

confidence: 99%

CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition

Hegde¹,

Valanarasu²,

Patel³

2023

Preprint

View full text Add to dashboard Cite

Figure 1: Overview of our proposed framework CLIP goes 3D (CG3D). We introduce a 3D Encoder in the CLIP framework and pre-train it using natural language supervision while also leveraging CLIP's pre-trained visual encoder. CG3D solves various practical tasks like zero-shot 3D recognition, 3D point cloud retrieval, scene querying with natural language, Moreover, it can serve as a strong initial weight for standard fine-tuning tasks.

show abstract

“…DeepAugment [23] was one of the first augmentation strategies to perform well on natural distribution shifts. Additionally, studies on CLIP-verse [39,29,33,18,35] have shown natural robustness. In our work, we take the best of both paradigms by leveraging the strengths of modern generative models to augment real datasets.…”

Section: Related Workmentioning

confidence: 99%

Leaving Reality to Imagination: Robust Classification via Generated Datasets

Bansal¹,

Grover²

2023

Preprint

View full text Add to dashboard Cite

Recent research on robustness has revealed significant performance gaps between neural image classifiers trained on datasets that are similar to the test set, and those that are from a naturally shifted distribution, such as sketches, paintings, and animations of the object categories observed during training. Prior work focuses on reducing this gap by designing engineered augmentations of training data or through unsupervised pretraining of a single large model on massive in-the-wild training datasets scraped from the Internet. However, the notion of a dataset is also undergoing a paradigm shift in recent years. With drastic improvements in the quality, ease-of-use, and access to modern generative models, generated data is pervading the web. In this light, we study the question: How do these generated datasets influence the natural robustness of image classifiers? We find that Imagenet classifiers trained on real data augmented with generated data achieve higher accuracy and effective robustness than standard training and popular augmentation strategies in the presence of natural distribution shifts. We analyze various factors influencing these results, including the choice of conditioning strategies and the amount of generated data. Lastly, we introduce and analyze an evolving generated dataset, ImageNet-G-v1, to better benchmark the design, utility, and critique of standalone generated datasets for robust and trustworthy machine learning. The code and datasets are available at https://github.com/Hritikbansal/ generative-robustness.

show abstract

CyCLIP: Cyclic Contrastive Language-Image Pretraining

Cited by 5 publications

References 0 publications

SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger

SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger

CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition

Leaving Reality to Imagination: Robust Classification via Generated Datasets

Contact Info

Product

Resources

About