How Fine-Tuning Allows for Effective Meta-Learning

Chua, Kurtland; Liu, Qi; Lee, Jason D.

doi:10.48550/arxiv.2105.02221

Cited by 5 publications

(7 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior works on transfer learning mainly analyze linear probing (Wu et al, 2020;Tripuraneni et al, 2020;Du et al, 2020). In recent work, (Chua et al, 2021) study regularized fine-tuning in an underparameterized regime where there is a unique global optimum. In contrast, our analysis studies the overparameterized regime (mirroring modern settings of zero train loss) where we need to analyze the trajectory of fine-tuning from the pretrained initialization because there is no unique optimizer of the objective function.…”

Section: Related Work and Discussionmentioning

confidence: 99%

Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution

Kumar¹,

Raghunathan²,

Jones³

et al. 2022

Preprint

View full text Add to dashboard Cite

When transferring a pretrained model to a downstream task, two popular methods are full fine-tuning (updating all the model parameters) and linear probing (updating only the last linear layer-the "head"). It is well known that fine-tuning leads to better accuracy in-distribution (ID). However, in this paper, we find that fine-tuning can achieve worse accuracy than linear probing out-of-distribution (OOD) when the pretrained features are good and the distribution shift is large. On 10 distribution shift datasets (Breeds-Living17, Breeds-Entity30, DomainNet, CIFAR → STL, CIFAR10.1, FMoW, ImageNetV2, ImageNet-R, ImageNet-A, ImageNet-Sketch), fine-tuning obtains on average 2% higher accuracy ID but 7% lower accuracy OOD than linear probing. We show theoretically that this tradeoff between ID and OOD accuracy arises even in a simple setting: fine-tuning overparameterized two-layer linear networks. We prove that the OOD error of fine-tuning is high when we initialize with a fixed or random head-this is because while fine-tuning learns the head, the lower layers of the neural network change simultaneously and distort the pretrained features. Our analysis suggests that the easy two-step strategy of linear probing then full fine-tuning (LP-FT), sometimes used as a fine-tuning heuristic, combines the benefits of both fine-tuning and linear probing. Empirically, LP-FT outperforms both fine-tuning and linear probing on the above datasets (1% better ID, 10% better OOD than full fine-tuning).

show abstract

Section: Related Work and Discussionmentioning

confidence: 99%

Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution

Kumar¹,

Raghunathan²,

Jones³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, our work focuses on the setting where all tasks share one common low-dimensional representation. [CLL21] shows the benefits of task-specific fine-tuning, which is fundamentally different from our refinement step. Our refinement step aims to reduce the representation dimension with slight information loss and help to improve the sample complexity of subsequent tasks.…”

Section: Related Workmentioning

confidence: 95%

Provable Lifelong Learning of Representations

Cao¹,

Liu²,

Vempala³

2021

Preprint

View full text Add to dashboard Cite

In lifelong learning, the tasks (or classes) to be learned arrive sequentially over time in arbitrary order. During training, knowledge from previous tasks can be captured and transferred to subsequent ones to improve sample efficiency. We consider the setting where all target tasks can be represented in the span of a small number of unknown linear or nonlinear features of the input data. We propose a provable lifelong learning algorithm that maintains and refines the internal feature representation. We prove that for any desired accuracy on all tasks, the dimension of the representation remains close to that of the underlying representation. The resulting sample complexity improves significantly on existing bounds. In the setting of linear features, our algorithm is provably efficient and the sample complexity for input dimension d, m tasks with k features up to error is Õ(dk 1.5 / + km/ ). We also prove a matching lower bound for any lifelong learning algorithm that uses a single task learner as a black box. Finally, we complement our analysis with an empirical study.

show abstract

“…This observation encourages the idea that there is some inductive bias relating to the initial weights of the model that might explain the success of fine-tuning. [20] and [21] both show that fine-tuned models generalize well when the representation used by the target task is approximately similar to the one used by the source tasks.…”

Section: Related Workmentioning

confidence: 99%

A Theoretical Analysis of Fine-tuning with Linear Teachers

Shachaf¹,

Brutzkus²,

Globerson³

2021

Preprint

View full text Add to dashboard Cite

Fine-tuning is a common practice in deep learning, achieving excellent generalization results on downstream tasks using relatively little training data. Although widely used in practice, it is lacking strong theoretical understanding. We analyze the sample complexity of this scheme for regression with linear teachers in several architectures. Intuitively, the success of fine-tuning depends on the similarity between the source tasks and the target task, however measuring it is non trivial. We show that a relevant measure considers the relation between the source task, the target task and the covariance structure of the target data. In the setting of linear regression, we show that under realistic settings a substantial sample complexity reduction is plausible when the above measure is low. For deep linear regression, we present a novel result regarding the inductive bias of gradient-based training when the network is initialized with pretrained weights. Using this result we show that the similarity measure for this setting is also affected by the depth of the network. We further present results on shallow ReLU models, and analyze the dependence of sample complexity there on source and target tasks. We empirically demonstrate our results for both synthetic and realistic data.

show abstract

How Fine-Tuning Allows for Effective Meta-Learning

Cited by 5 publications

References 23 publications

Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution

Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution

Provable Lifelong Learning of Representations

A Theoretical Analysis of Fine-tuning with Linear Teachers

Contact Info

Product

Resources

About