Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights

Abstract

This work presents a method for adapting a single, fixed deep neural network to multiple tasks without affecting performance on already learned tasks. By building upon ideas from network quantization and pruning, we learn binary masks that "piggyback" on an existing network, or are applied to unmodified weights of that network to provide good performance on a new task. These masks are learned in an end-toend differentiable fashion, and incur a low overhead of 1 bit per network parameter, per task. Even though the underlying network is fixed, the ability to mask individual weights allows for the learning of a large number of filters. We show performance comparable to dedicated fine-tuned networks for a variety of classification tasks, including those with large domain shifts from the initial task (ImageNet), and a variety of network architectures. Unlike prior work, we do not suffer from catastrophic forgetting or competition between tasks, and our performance is agnostic to task ordering. Fig. 1: Overview of our method for learning piggyback masks for fixed backbone networks. During training, we maintain a set of real-valued weights m r which are passed through a thresholding function to obtain binary-valued masks m. These masks are applied to the weights W of the backbone network in an elementwise fashion, keeping individual weights active, or masked out. The gradients obtained through backpropagation of the task-specific loss are used to update the realvalued mask weights. After training, the real-valued mask weights are discarded and only the thresholded mask is retained, giving one network mask per task.

NetTailor: Tuning the Architecture, Not Just the Weights

“…The first protocol described in [38] consists of five datasets: CUB [67], Stanford Cars [25], Oxford Flowers [46], WikiArt [56] and Sketch [10]. Following [38], we use the same train/test set splits, and apply the NETTAILOR procedure to the same backbone network, ResNet50, with an input size 224x224. The second protocol is the visual decathlon benchmark [49] and consists of ten different datasets including ImageNet, Omniglot, German Traffic Signs, among others.…”

“…While feature extraction shares most weights across datasets, differences between the source and target domains cannot be corrected, thus achieving low performance. More refined methods, such as PackNet [39] and Piggyback [38], try to selectively adjust the network weights in order to remember previous tasks, or freeze the backbone network and learn a small set of task-specific parameters (a set of masking weights in the case of Piggyback) that is used to bridge the gap between source and target tasks. All these methods ignore the fact that source and target datasets can differ in terms of difficulty, and thus the architecture itself should be adjusted to the target task, not just the weights.…”

Single-Net Continual Learning with Progressive Segmented Training

“…Figure 2 The weight distribution after this step is shown in Figure 3d. It is worth mentioning that previous works, such as PackNet [22] and Piggyback [21], prune the secondary parameters and thus, distort the weight distribution (Figure 3b). At epoch 300, task T 2 appears and updates the parameters.…”

“…In this case, the rest of the parameters no longer contain prior knowledge, violating the aforementioned properties of an ideal continual learning system. For instance, PackNet [21] and Piggyback [22] achieve strong performance on multi-head evaluation but not on single-head.…”

“…These important parameters are kept frozen and will not be updated by future tasks. Different from [22] and [21], the secondary parameters (in light blue) are NOT pruned in PST. Instead, new tasks will start from secondary parameters and update the network, which is essential to achieve single-head classification.…”