Learning to Prompt for Vision-Language Models

Abstract

Vision-language pre-training has recently emerged as a promising alternative for representation learning. It shifts from the tradition of using images and discrete labels for learning a fixed set of weights, seen as visual concepts, to aligning images and raw text for two separate encoders. Such a paradigm benefits from a broader source of supervision and allows zero-shot transfer to downstream tasks since visual concepts can be diametrically generated from natural language, known as prompt. In this paper, we identify that a major challenge of deploying such models in practice is prompt engineering. This is because designing a proper prompt, especially for context words surrounding a class name, requires domain expertise and typically takes a significant amount of time for words tuning since a slight change in wording could have a huge impact on performance. Moreover, different downstream tasks require specific designs, further hampering the efficiency of deployment. To overcome this challenge, we propose a novel approach named context optimization (CoOp). The main idea is to model context in prompts using continuous representations and perform end-to-end learning from data while keeping the pre-trained parameters fixed. In this way, the design of task-relevant prompts can be fully automated. Experiments on 11 datasets show that CoOp effectively turns pre-trained vision-language models into data-efficient visual learners, requiring as few as one or two shots to beat hand-crafted prompts with a decent margin and able to gain significant improvements when using more shots (e.g., at 16 shots the average gain is around 17% with the highest reaching over 50%). CoOp also exhibits strong robustness to distribution shift.

Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks

“…However, the performance gap from full-model fine-tuning closes up as the pre-trained model gets larger [33,42]. Inspired by the success of prompt tuning in NLP, [77] applies prompt tuning to visual-linguistic pre-trained models (e.g., CLIP [55]) to perform few-shot image classification. [51,58] further apply a residual feature adapter to improve the few-shot performance.…”

Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts

“…Continuous prompts. There is a line of work focused on tuning continuous prompts (Li and Liang, 2021;Lester et al, 2021;Zhong et al, 2021;Qin and Eisner, 2021;Zhou et al, 2021). A recurring theme in this line of work is the strength of continuous prompt in results in strong, yet compact models-compared to conventional architecture fine-tuning approaches.…”

VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts

“…Inspired by the success of pre-train models [6], [4] [25] and SimVLM [37] use attention architecture further improve the performance of vision-language tasks. At present, the recent breakthrough in vision-language learning, particularly CLIP [31] and ALIGN [18] are driven by the noisy largescale datasets available in the Internet, which is 400 million image-text pair for CLIP and 1.8 billion noisy image-text pairs for ALIGN.To finetune vision-Language Models on downstream tasks like few-shot classification task, CoOp [42] propose to learn soft prompts represented by continuous context vectors as alternative for hand-craft prompt while CLIP-Adapter propose to adopts an additional bottleneck layer to learn new features and performs residual style feature blending with the original pre-trained features. Though CoOp and CLIP-Adapter achieve significant performance in the perspective of prompt learning and feature adapters, our VT-CLIP explores the impact of instance-level visual feature on text feature with a cross-attention module.…”