CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP

Abstract

Contrastive learning with the InfoNCE objective is exceptionally successful in various self-supervised learning tasks. Recently, the CLIP model yielded impressive results on zero-shot transfer learning when using InfoNCE for learning visual representations from natural language supervision. However, InfoNCE as a lower bound on the mutual information has been shown to perform poorly for high mutual information. In contrast, the InfoLOOB upper bound (leave one out bound) works well for high mutual information but suffers from large variance and instabilities. We introduce "Contrastive Leave One Out Boost" (CLOOB), where modern Hopfield networks boost learning with the InfoLOOB objective. Modern Hopfield networks replace the original embeddings by retrieved embeddings in the InfoLOOB objective. The retrieved embeddings give InfoLOOB two assets. Firstly, the retrieved embeddings stabilize InfoLOOB, since they are less noisy and more similar to one another than the original embeddings. Secondly, they are enriched by correlations, since the covariance structure of embeddings is reinforced through retrievals. We compare CLOOB to CLIP after learning on the Conceptual Captions and the YFCC dataset with respect to their zero-shot transfer learning performance on other datasets. CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.

Conditional Prompt Learning for Vision-Language Models

“…Recent vision-language models [13,24,33,40] bridge the two modalities by learning two encoders jointly. Also, the models are now built with much larger neural networks.…”

“…After consuming 400 million data pairs, the CLIP model demonstrates a remarkable zero-shot image recognition capability. Similar to CoOp [62], our approach is orthogonal to the research of CLIP-like models [13,24,33,40], aiming to offer an efficient solution for adapting pre-trained vision-language models to downstream applications.…”

“…Recent research in large-scale vision-language pretraining has achieved striking performance in zero-shot image recognition [13,24,33,40], demonstrating a potential in learning open-world visual concepts for such a paradigm. The key design lies in how visual concepts are modeled.…”

Robust Cross-Modal Representation Learning with Progressive Self-Distillation

“…b. Vision-Language Pretraining: Joint vision-language pretraining (VLP) is an active research area [1,19,39,63,68] where the availability of large-scale image-text datasets e.g., YFCC100M [71] and Conceptual Captions [9,67] has played a key role in its progress. Although multiple concurrent works are being proposed to further improve VLP models [75], our work is different from them in a few important ways.…”

“…The convergence of self-supervised pretraining techniques in natural language processing and computer vision have brought about a renaissance of cross-modal representation learning methods [1,19,30,39,52,63,68,75] where largescale weakly correlated multimodal data (e.g., image-text pairs) is used to learn cross-modal representations using contrastive learning techniques. In particular, the recently proposed CLIP [63] model has garnered significant attention due to its impressive zero-shot recognition ability and excellent transfer performance on downstream tasks.…”