When do Models Generalize? A Perspective from Data-Algorithm Compatibility

Abstract

Benign overfitting demonstrates that overparameterized models can perform well on test data while fitting noisy training data. However, it only considers the final min-norm solution in linear regression, which ignores the algorithm information and the corresponding training procedure. In this paper, we generalize the idea of benign overfitting to the whole training trajectory instead of the min-norm solution and derive a time-variant bound based on the trajectory analysis. Starting from the timevariant bound, we further derive a time interval that suffices to guarantee a consistent generalization error for a given feature covariance. Unlike existing approaches, the newly proposed generalization bound is characterized by a time-variant effective dimension of feature covariance. By introducing the time factor, we relax the strict assumption on the feature covariance matrix required in previous benign overfitting under the regimes of overparameterized linear regression with gradient descent. This paper extends the scope of benign overfitting, and experiment results indicate that the proposed bound accords better with empirical evidence.

Provable Generalization of Overparameterized Meta-learning Trained with SGD

“…Overparameterized linear model. There has been several recent progress in theoretical understanding of overparameterized linear model under different scenarios, where the main goal is to provide non-asymptotic generalization guarantees, such as studies of linear regression [5], ridge regression [36], constant-stepsize SGD [44], decaying-stepsize SGD [41], GD [42], Gaussian Mixture models [39]. This paper aims to derive the non-asymptotic excess risk bound for MAML under mixed linear model, which can be independent of data dimension d and still converge as the iteration T enlarges.…”