A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision

Tejankar, Ajinkya; Sanjabi, Maziar; Wu, Bichen; Xie, Sihong; Khabsa, Madian; Pirsiavash, Hamed; Firooz, Hamed

doi:10.48550/arxiv.2112.13884

Cited by 2 publications

(2 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the vision side, show that a bag-of-local-features model performs almost as well as their state-of-the-art counterparts. Closer to our experiments, the work of Tejankar et al (2021) shows that training contrastive vision language models using only bag-of-words in place of the caption does not significantly hurt performance on zero-shot classification. Our work generalize these results, showcasing the general limits of vision language models when dealing with relations, attributes and shuffled captions.…”

Section: Related Worksupporting

confidence: 47%

When and why vision-language models behave like bags-of-words, and what to do about it?

Yüksekgönül¹,

Bianchi²,

Kalluri³

et al. 2022

Preprint

View full text Add to dashboard Cite

Despite the success of large vision and language models (VLMs) in many downstream applications, it is unclear how well they encode the compositional relationships between objects and attributes. Here, we create the Attribution, Relation, and Order (ARO) benchmark to systematically evaluate the ability of VLMs to understand different types of relationships, attributes, and order information. ARO consists of Visual Genome Attribution, to test the understanding of objects' properties; Visual Genome Relation, to test for relational understanding; and COCO-Order & Flickr30k-Order, to test for order sensitivity in VLMs. ARO is orders of magnitude larger than previous benchmarks of compositionality, with more than 50,000 test cases. We present the settings where state-of-the-art VLMs behave like bagsof-words-i.e. when they have poor relational understanding, can blunder when linking objects to their attributes, and demonstrate a severe lack of order sensitivity. VLMs are predominantly trained and evaluated on large scale datasets with rich compositional structure in the images and captions. Yet, training on these datasets has not been enough to address the lack of compositional understanding, and evaluating on these datasets has failed to surface this deficiency. To understand why these limitations emerge and are not represented in the standard tests, we zoom into the evaluation and training procedures. We demonstrate that it is possible to perform well on image-text retrieval over existing datasets without using the composition and order information. This further motivates the value of using ARO to benchmark VLMs. Given that contrastive pretraining optimizes for retrieval on large datasets with similar shortcuts, we hypothesize that this can explain why the models do not need to learn to represent compositional information. This finding suggests a natural solution: composition-aware hard negative mining. We show that a simple-to-implement modification of contrastive learning significantly improves the performance on tasks requiring understanding of order and compositionality.

show abstract

Section: Related Worksupporting

confidence: 47%

When and why vision-language models behave like bags-of-words, and what to do about it?

Yüksekgönül¹,

Bianchi²,

Kalluri³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…While it has been shown that poisoning web-scale datasets such as CC3M is practical [5], we assume that the version of CC3M we downloaded in January 2022 is clean. Although CC3M is smaller in size than the 400 million pairs used to train the original CLIP model [42], it is suitable for our storage and computational resources and has been used in multiple language-image pretraining studies [6,28,34,47,18]. Like in [42], we use a ResNet-50 model as the CLIP vision encoder and a transformer as the text encoder.…”

Section: Clip Pretrainingmentioning

confidence: 99%

CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning

Bansal¹,

Singhi²,

Yu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Multimodal contrastive pretraining has been used to train multimodal representation models, such as CLIP, on large amounts of paired image-text data. However, previous studies have revealed that such models are vulnerable to backdoor attacks. Specifically, when trained on backdoored examples, CLIP learns spurious correlations between the embedded backdoor trigger and the target label, aligning their representations in the joint embedding space. Injecting even a small number of poisoned examples, such as 75 examples in 3 million pretraining data, can significantly manipulate the model's behavior, making it difficult to detect or unlearn such correlations. To address this issue, we propose CleanCLIP, a finetuning framework that weakens the learned spurious associations introduced by backdoor attacks by independently re-aligning the representations for individual modalities. We demonstrate that unsupervised finetuning using a combination of multimodal contrastive and unimodal self-supervised objectives for individual modalities can significantly reduce the impact of the backdoor attack. Additionally, we show that supervised finetuning on task-specific labeled image data removes the backdoor trigger from the CLIP vision encoder. We show empirically that CleanCLIP maintains model performance on benign examples while erasing a range of backdoor attacks on multimodal contrastive learning. * Equal Contribution † Equal Contribution ‡ Equal Advising Preprint. Under review.

show abstract

A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision

Cited by 2 publications

References 31 publications

When and why vision-language models behave like bags-of-words, and what to do about it?

When and why vision-language models behave like bags-of-words, and what to do about it?

CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning

Contact Info

Product

Resources

About