Prompt-aligned Gradient for Prompt Tuning

Abstract

Thanks to the large pre-trained vision-language models (VLMs) like CLIP [36], we can craft a zero-shot classifier by "prompt", e.g., the confidence score of an image being "[CLASS]" can be obtained by using the VLM provided similarity measure between the image and the prompt sentence "a photo of a [CLASS]". Therefore, prompt shows a great potential for fast adaptation of VLMs to downstream tasks if we fine-tune the prompt-based similarity measure. However, we find a common failure that improper fine-tuning may not only undermine the prompt's inherent prediction for the task-related classes, but also for other classes in the VLM vocabulary. Existing methods still address this problem by using traditional anti-overfitting techniques such as early stopping and data augmentation, which lack a principled solution specific to prompt. We present Prompt-aligned Gradient, dubbed ProGrad, to prevent prompt tuning from forgetting the the general knowledge learned from VLMs. In particular, ProGrad only updates the prompt whose gradient is aligned (or non-conflicting) to the "general direction", which is represented as the gradient of the KL loss of the pre-defined prompt prediction. Extensive experiments demonstrate the stronger few-shot generalization ability of ProGrad over state-of-the-art prompt tuning methods. Codes are available at https://github.com/BeierZhu/Prompt-align.

StyLIP: Multi-Scale Style-Conditioned Prompt Learning for CLIP-based Domain Generalization

“…However, using tokens referring to the domain identifiers in the prompts improves baseline CLIP's performance. Finally, StyLIP outperforms the previous best prompting techniques [8,9] substantially, highlighting the importance of style and content disentanglement in the prompts for DG tasks. manner.…”

“…In this paper, we follow this research trend, but different from existing prompting methods [3][4][5]8], which evaluate the generalization capabilities of CLIP on datasets where the domain shift is limited (e.g. variants of ImageNet [17]), we study a more challenging setting where the visual appearance of images varies significantly from different domains.…”

“…1). However, such approaches are suboptimal for DG, as it is unclear whether the prompts refined from random vectors [4,8] can encode the domain knowledge effectively. In this respect, Zhang et al [19] proposes a domain-prompt initialization strategy from the batch statistics of the visual features, but [19] considers domain-level supervision and overlooks to capture important lower-level style characteristics.…”

Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models

“…For example, CLIP [81] adopts linear probing [12,31,32,109] and full-finetuning [25,31,48,99,101,109] when transferring to downstream tasks. Prompt adaptation of CLIP [63,81,105,112,114] is motivated by the success of prefix-tuning for language models [16,22,30,45,61,78,84,85,89]. Similarly, CLIP-Adapter [21] and Tip-Adapter [111] are inspired by parameter-efficient finetuning methods [39,44,110] that optimize lightweight MLPs while freezing the encoder.…”

“…In contrast to our cross-modal approach, most prior works simply follow the popular practice of finetuning uni-modal foundation models, such as large vision [12,31,32] or language models [8,17,62]. For example, CoOp [113] and other prompting methods [63,112,114] finetune CLIP via prefix tuning to replace hand-engineered prompts such as "a photo of a {cls}" with learned word tokens. Similarly, inspired by parameter-efficient tuning of language models [39], adapter-based methods [21,111] finetune CLIP by inserting lightweight multi-layer-perceptrons (MLPs).…”