Training Neural Networks with Fixed Sparse Masks

Abstract

During typical gradient-based training of deep neural networks, all of the model's parameters are updated at each iteration. Recent work has shown that it is possible to update only a small subset of the model's parameters during training, which can alleviate storage and communication requirements. In this paper, we show that it is possible to induce a fixed sparse mask on the model's parameters that selects a subset to update over many iterations. Our method constructs the mask out of the k parameters with the largest Fisher information as a simple approximation as to which parameters are most important for the task at hand. In experiments on parameter-efficient transfer learning and distributed training, we show that our approach matches or exceeds the performance of other methods for training with sparse updates while being more efficient in terms of memory usage and communication costs. We release our code publicly to promote further applications of our approach. 2 * Equal contribution. 2 Code for our work can be found at https://github.com/varunnair18/FISH.

Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning

“…FishMask [26] The Fisher is first computed on the training examples and we keep 0.2% or 0.02% of the parameters. Then, these parameters are trained for 1500 steps with a learning rate of 3e −4 .…”

“…Early methods proposed adding adapters [22][23][24], which are small feed-forward networks inserted between the layers in the pre-trained model whose parameters are updated during fine-tuning while the remainder of the pre-trained model is left fixed. Since then, various sophisticated PEFT methods have been proposed, including methods that choose a sparse subset of parameters to train [25,26], produce low-rank updates [13], perform optimization in a lower-dimensional subspace [27], add low-rank adapters using hypercomplex multiplication [28], and more. Relatedly, prompt tuning [14] concatenates learned continuous embeddings to the model's input to induce it to perform a task and can be seen as a PEFT method [29].…”

“…To validate (IA) 3 , we compare it to a large variety of existing adaptation methods in our setting of fine-tuning T0-3B on few-shot datasets from held-out tasks. Specifically, we compare against eight strong baseline methods: BitFit [47] which updates only the bias parameters; Adapters [23] which introduce task-specific layers after the self-attention and position-wise feed-forward networks; Compacter and Compacter++ [28] which improve upon adapters by using low-rank matrices and hypercomplex multiplication; prompt tuning [14] which learns task-specific prompt embeddings that are concatenated to the model's input; FISH Mask [26] which chooses a subset of parameters to update based on their approximate Fisher information; Intrinsic SAID [27] which performs optimization in a low-dimensional subspace; and LoRA [13] which assigns low-rank updates to parameter matrices. Additionally, we include the simple baselines of full-model fine-tuning and updating only the layer normalization parameters.…”

“…Prompt tuning and other PEFT methods have also been explored outside of the context of language models (e.g. vision [22,69] and vision-and-language models [26]).…”

Deep learning via message passing algorithms based on belief propagation

“…We call sparsity the fraction of zeroed weights. This kind of non-adaptive pruning is known to largely hinder learning (Frankle et al 2021, Sung et al 2021. In the right panel of figure 2, we report results on sparse binary networks in which we train a MLP with 2 hidden layers of 101 units on the MNIST dataset.…”