Prompt Generation Networks for Efficient Adaptation of Frozen Vision Transformers

Abstract

Large-scale pretrained models, especially those trained from vision-language data have demonstrated the tremendous value that can be gained from both larger training datasets and models. Thus, in order to benefit from these developments, there is renewed interest in transfer learning and adapting models from large-scale general pretraining to particular downstream tasks. However, the continuously increasing size of the models means that even the classic approach of finetuning is becoming infeasible for all but big institutions. Prompt leaning has emerged as a flexible way to adapt models by solely learning additional inputs to a model that is kept frozen, but so far performances remained inferior to finetuning. To address this, we propose the Prompt Generation Network (PGN) that generates input-dependent prompts by sampling from a learned library of tokens. We show the PGN is effective in adapting pretrained models to various new datasets. It surpasses previous prompt-learning methods by a large margin and even fullfinetuning on 5 out of 12 datasets while requiring 100x less parameters. PGN can even be used for training and inferring on multiple datasets simultaneously and learns to allocate tokens between domains. Given these findings, we conclude that PGN is a viable and scalable approach for downstream adaptation of frozen models. Code is available at https://github.com/jochemloedeman/PGN.

Rethinking Visual Prompt Learning as Masked Visual Token Modeling

“…Experimentally, VPTM outperforms other visual prompt learning methods [26,1,6,38,57] with better efficiency. Extensive experiments show the consistency between pretraining and downstream visual classification contributes to the robustness against learning strategies for different datasets, prompt locations, prompt length, and prototype dimensions.…”

“…Mid. Visual prompt methods designed on discriminative pre-trained models concentrate on adding prompts to input space (VPT [26], VP [1], ILM-VP [6], EVP [57]) or learning prompt network (PGN [38]), while ignoring task consistency. Bottom.…”

“…Witnessing the success of prompt learning in NLP, researchers introduce prompt learning into vision applications [41,38,54]. VPT [26] prepends a few parameters as prompts to the input sequence of ViT [15], which has been supervised pre-trained on ImageNet-21k [13] in a discriminative way.…”

“…Visual prompting (VP) [1] modifies the pixel space with learnable parameters to perform visual prompt learning on CLIP [46], which has been pre-trained by contrastive learning. Till now, the current visual prompting methods [38,54,57,41] are all designed on discriminative pre-trained models shown in the middle of Fig. 1.…”

“…There lacks prompt learning method carefully designed for the generative pre-trained visual model. Particularly, regardless of the efforts paid on adding prompts in the input space [26,1,57,6], learning prompt networks [38] or designing prompt blocks [41], unifying the forms of pretraining and downstream applications by task reformulation to achieve consistency remains unexplored. In view of the improved performance, efficiency and stability brought by the task consistency of prompt learning in NLP, we aim at generative visual prompt learning by inheriting the generative pre-training task to achieve consistency.…”