AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning

Abstract

With the rapid adoption of machine learning (ML), a number of domains now use the approach of fine-tuning models pre-trained on a large corpus of data. However, our experiments show that even fine-tuning on models like BERT can take many hours when using GPUs. While prior work proposes limiting the number of layers that are fine-tuned, e.g., freezing all layers but the last layer, we find that such static approaches lead to reduced accuracy. We propose, AutoFreeze, a system that uses an adaptive approach to choose which layers are trained and show how this can accelerate model fine-tuning while preserving accuracy. We also develop mechanisms to enable efficient caching of intermediate activations which can reduce the forward computation time when performing fine-tuning. Our evaluation on four NLP tasks shows that AutoFreeze, with caching enabled, can improve fine-tuning performance by up to 2.55×.

Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning

“…Radiya-Dixit and Wang [20] showed that it suffices to fine-tune only the most critical layers. Similar results can be found in [15,22]. While a partial FT can be advantageous over a full FT, they both require the pre-trained BERT to be modified.…”

Improving fine-tuning of self-supervised models with Contrastive Initialization

“…Shen et al [45] observe that not all pre-trained knowledge is necessarily beneficial for few-shot learning tasks, and hence propose to only transfer partial knowledge from the pre-trained model by selectively freezing some layers and fine-tuning others. SpotTune [46] and AutoFreeze [47] are adaptive fine-tuning approaches that also freeze some layers while adapting others, resulting in improved performance on computer vision and language modeling respectively. Recently, Lee et al [25] show that surgical fine-tuning (i.e., fine-tuning only a subset of layers) can yield better results than end-to-end fine-tuning when adapting to various distribution shifts.…”

HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks

“…These methods generally include two categories according to whether new trainable parameters are introduced. One category is that only a subset of model parameters can be updated while freezing the remain (Liu et al, 2021b;Lee et al, 2019). The other is introducing a few task-specific new parameters to different parts of pretrained models, such as before multi-head attention (Li & Liang, 2021), after feedforward layers (Houlsby et al, 2019) or Mixed-and-Match methods (MAM adapter) proposed by He et al (2021).…”

“…To mitigate these issues, there has recently been one line of research on Parameter-Efficient Language model Tuning (PELT). A few lightweight transfer learning methods have been proposed and they only update a subset of model parameters while freeze the remaining most parameters (Liu et al, 2021b). Extra trainable task-specific model parameters can also be newly introduced to PLMs, such as the widely used adapter-tuning (Houlsby et al, 2019) and prefixtuning (Li & Liang, 2021) methods.…”