Probing transfer learning with a model of synthetic correlated datasets

Abstract

Transfer learning can significantly improve the sample efficiency of neural networks, by exploiting the relatedness between a data-scarce target task and a data-abundant source task. Despite years of successful applications, transfer learning practice often relies on ad-hoc solutions, while theoretical understanding of these procedures is still limited. In the present work, we re-think a solvable model of synthetic data as a framework for modeling correlation between data-sets. This setup allows for an analytic characterization of the generalization performance obtained when transferring the learned feature map from the source to the target task. Focusing on the problem of training two-layer networks in a binary classification setting, we show that our model can capture a range of salient features of transfer learning with real data. Moreover, by exploiting parametric control over the correlation between the two data-sets, we systematically investigate under which conditions the transfer of features is beneficial for generalization.

Inducing bias is simpler than you think

“…In Fig. 5, we show that a positive transfer effect [55] can be traced between the two groups when the rules are sufficiently similar. In fact, the performance on the smaller group only deteriorates if the dataset is split.…”

Data-Efficient Generation of Protein Conformational Ensembles with Backbone-to-Side-Chain Transformers

“…While this approach is appealing, designing a transfer learning strategy that does not over-constrain the model and result in negative transfer can be challenging. 23,24 Second, building a model that incorporates strong physical inductive biases in the form of constraints or preconditioners can improve generalization by ensuring that generated samples respect the imposed constraints. 35,73 Physics-based bottom-up coarse-grained models 28,29,51,58 offer a particularly attractive approach to building such a prior.…”

“…To construct models that do not require task-specific data, there are two dominant strategies: first, one can use data from a closely related task that is inexpensive to acquire and carry out transfer learning. While this approach is appealing, designing a transfer learning strategy that does not overconstrain the model and result in negative transfer can be challenging. , Second, building a model that incorporates strong physical inductive biases in the form of constraints or preconditioners can improve generalization by ensuring that generated samples respect the imposed constraints. , Physics-based bottom-up coarse-grained models ,,, offer a particularly attractive approach to building such a prior . At the same time, imposing these restrictions may impact the trainability and expressiveness of the model in ways that are unproductive .…”

Exact learning dynamics of deep linear networks with prior knowledge *

“…For instance, having learned about the attributes of nine animals, we may learn about the tenth more quickly (McClelland et al 1995, Murphy 2004, McClelland 2013, Flesch et al 2018. In machine learning, the impact of prior knowledge on learning is evident in a range of paradigms including reversal learning (Erdeniz and Atalay 2010), transfer learning (Taylor and Stone 2009, Thrun and Pratt 2012, Lampinen and Ganguli 2018, Gerace et al 2022, continual learning (Kirkpatrick et al 2017, Zenke et al 2017, Parisi et al 2019, curriculum learning (Bengio et al 2009), and meta learning (Javed and White 2019). One form of prior knowledge in deep networks is the initial network state, which is known to strongly impact learning dynamics (Saxe et al 2014, Pennington et al 2017 Figure 1.…”