HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks

Abstract

The workflow of pretraining and fine-tuning has emerged as a popular paradigm for solving various NLP and V&L (Vision-and-Language) downstream tasks. With the capacity of pretrained models growing rapidly, how to perform parameterefficient fine-tuning has become fairly important for quick transfer learning and deployment. In this paper, we design a novel unified parameterefficient transfer learning framework that works effectively on both pure language and V&L tasks. In particular, we use a shared hypernetwork that takes trainable hyper-embeddings as input, and outputs weights for fine-tuning different small modules in a pretrained language model, such as tuning the parameters inserted into multi-head attention blocks (i.e., prefix-tuning) and feedforward blocks (i.e., adapter-tuning). We define a set of embeddings (e.g., layer, block, task and visual embeddings) as the key components to calculate hyper-embeddings, which thus can support both pure language and V&L tasks. Our proposed framework adds fewer trainable parameters in multi-task learning while achieves superior performances and transfer ability compared to state-of-the-art methods. Empirical results on the GLUE benchmark and multiple V&L tasks confirm the effectiveness of our framework on both textual and visual modalities. 1 * Equal contribution 1 Work is done at the internship of Noah's Ark Lab, Huawei Technologies.

Parameter-Efficient Methods for Metastases Detection fromClinical Notes

“…For example, SPoT [15] suggests initializing the downstream task prompts with prompts that have been tuned on a mixture of related tasks. Meanwhile, HyperPELT [16] trains a hypernetwork that generates trainable parameters for the main model, including prompt tokens. Another approach, ATTEMPT [17], learns prompts for all the source tasks and then creates an instance-wise prompt for the target task by combining the source tasks' prompts and a newly initialized prompt using an attention block.…”