Towards a Unified View of Parameter-Efficient Transfer Learning

Abstract

Fine-tuning large pretrained language models on downstream tasks has become the de-facto learning paradigm in NLP. However, conventional approaches finetune all the parameters of the pretrained model, which becomes prohibitive as the model size and the number of tasks grow. Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance. While effective, the critical ingredients for success and the connections among the various methods are poorly understood. In this paper, we break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them. Specifically, we re-frame them as modifications to specific hidden states in pretrained models, and define a set of design dimensions along which different methods vary, such as the function to compute the modification and the position to apply the modification. Through comprehensive empirical studies across machine translation, text summarization, language understanding, and text classification benchmarks, we utilize the unified view to identify important design choices in previous methods. Furthermore, our unified framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efficient fine-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks. * Equal Contribution. Order determined by random dice rolling.

A Roadmap for Big Model

“…Although each of these three approaches has its own focus, the central idea is to keep the pre-trained parameters constant while training lightweight alternatives to achieve adaptation for downstream tasks. There have also been some recent attempts to grasp the internal connection of these strategies and build a unified parameter-efficient tuning framework [333,334].…”

Combining Modular Skills in Multitask Learning

“…Moreover, we show that our method can be used in tandem with several parameter-efficient methods (He et al, 2021) in order to make the increase in time and space complexity due to skill-specific parameters negligible. In particular, we explore sparse adaptation with Lottery-Ticket Sparse Fine-Tuning (LT-SFT; Ansell et al, 2022) and low-rank adaptation with Low-Rank Adapters (LoRA;Hu et al, 2021).…”

Continuous Detection, Rapidly React: Unseen Rumors Detection based on Continual Prompt-Tuning

“…Adapter: Freeze the pre-trained model and train a residual Adapter (Houlsby et al, 2019). ParallelAdapter: A variant by transferring the parallel insertion of prefix tuning into adapters (He et al, 2021). prompt-tuning (CLS/VER): which only tunes soft-prompts with a frozen language model (Lester et al, 2021), prompt for transformer's first layer.…”

“…The level of catastrophic forgetting in EANN (Wang et al, 2018) is somewhat reduced compared to Fine-tuning but is still severe. Prompt-tuning and p-tuning v2 are somewhat related to the adapter method in the form of parameter tuning (He et al, 2021), but their performance in CL differs. The prompt-based model is better than the adapter in both datasets.…”