What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning

Abstract

Pretrained transformer-based language models have achieved state of the art across countless tasks in natural language processing. These models are highly expressive, comprising at least a hundred million parameters and a dozen layers. Recent evidence suggests that only a few of the final layers need to be fine-tuned for high quality on downstream tasks. Naturally, a subsequent research question is, "how many of the last layers do we need to fine-tune?" In this paper, we precisely answer this question. We examine two recent pretrained language models, BERT and RoBERTa, across standard tasks in textual entailment, semantic similarity, sentiment analysis, and linguistic acceptability. We vary the number of final layers that are fine-tuned, then study the resulting change in task-specific effectiveness. We show that only a fourth of the final layers need to be fine-tuned to achieve 90% of the original quality. Surprisingly, we also find that fine-tuning all layers does not always help.

AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning

“…We observe that we gain similar speedup and max accuracy for all the runs for each dataset. We also include the test accuracy convergence curve with respect to time for each of the three repeated runs using stepped learning rate schedule for each dataset in Figures 19,20,21,22,23,and 24. We see that AutoFreeze and full fine-tuning achieve comparable max accuracy with an average end-to-end training speedup of 2.05×, 1.55×, 2.05×, 1.94×, 1.81×, and 1.56× for AG News, Sogou News, IMDb, Yelp F., SQuAD2.0 and SWAG respectively. We can also see that the freezing speedup is on the same scale across different runs.…”

“…One direct approach to reduce the computational cost is to only fine-tune a subset of the layers [22]. For example, as shown in Figure 2, only updating the last k layers of the BERT can lead to an almost linear decrease in time taken per iteration.…”

“…First we consider simple static freezing schemes where a fixed number of layers are chosen to be updated during training as presented in [22]. In Figure 4 we compare static freezing schemes when fine-tuning BERT BASE with IMDb and Sogou dataset.…”

“…A natural approach to improve fine-tuning performance is to limit the number of layers of the model that are updated, thus making it similar to transfer learning. For example, if we consider BERT BASE which has 12 encoding blocks, prior approach in [22] trains a fixed number of blocks (e.g., the last 4 blocks) and freezes the weights for the remaining blocks. However, this approach can affect the final model accuracy.…”

Efficient DNN Training with Knowledge-Guided Layer Freezing

“…We observe that in transfer learning, freezing layers is mainly used for solving the overfitting problem [24]. While techniques such as static freezing [49] and cosine annealing [12] can reduce backward computation cost, accuracy loss is a common side effect. Thus, the main challenge of extending layer freezing to generic DNN training is how to maintain accuracy by only freezing the converged layers.…”