Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning

Abstract

Transfer-learning methods aim to improve performance in a data-scarce target domain using a model pretrained on a data-rich source domain. A cost-efficient strategy, linear probing, involves freezing the source model and training a new classification head for the target domain. This strategy is outperformed by a more costly but state-of-the-art method-fine-tuning all parameters of the source model to the target domain-possibly because fine-tuning allows the model to leverage useful information from intermediate layers which is otherwise discarded by the later pretrained layers. We explore the hypothesis that these intermediate layers might be directly exploited. We propose a method, Head-to-Toe probing (HEAD2TOE), that selects features from all layers of the source model to train a classification head for the target-domain. In evaluations on the Visual Task Adaptation Benchmark (VTAB), Head2Toe matches performance obtained with fine-tuning on average while reducing training and storage cost hundred folds or more, but critically, for out-of-distribution transfer, Head2Toe outperforms fine-tuning. 1

Continual Learning with Foundation Models: An Empirical Study of Latent Replay

“…8 we show the improvement achieved through ensembling the final output with each of the intermediate layers over using the final layer only (with the ViT-B/16 encoder). Similar to Evci et al (2022), we show that there always exists a hidden layer ensembling which always leads to a slight improvement of the CL accuracy (e.g. 2.5%-points on FGVCAircraft).…”