Learning-to-Learn Stochastic Gradient Descent with Biased Regularization

Abstract

We study the problem of learning-to-learn: inferring a learning algorithm that works well on tasks sampled from an unknown distribution. As class of algorithms we consider Stochastic Gradient Descent on the true risk regularized by the square euclidean distance to a bias vector. We present an average excess risk bound for such a learning algorithm. This result quantifies the potential benefit of using a bias vector with respect to the unbiased case. We then address the problem of estimating the bias from a sequence of tasks. We propose a meta-algorithm which incrementally updates the bias, as new tasks are observed. The low space and time complexity of this approach makes it appealing in practice. We provide guarantees on the learning ability of the meta-algorithm. A key feature of our results is that, when the number of tasks grows and their variance is relatively small, our learning-tolearn approach has a significant advantage over learning each task in isolation by Stochastic Gradient Descent without a bias term. We report on numerical experiments which demonstrate the effectiveness of our approach.

Provable Meta-Learning of Linear Representations

“…There has also been line of recent work providing guarantees for gradient-based meta-learning (MAML) [Finn et al, 2017]. Finn et al [2019], Khodak et al [2019a] and Denevi et al [2019] work in the framework of online convex optimization (OCO) and use a notion of task similarity that assumes closeness of all tasks to a single fixed point in parameter space to provide guarantees. Khodak et al [2019b] strengthens earlier meta-learning guarantees in the OCO framework and provides bounds with more general notions of data-dependent task similarity.…”

Meta-strategy for Learning Tuning Parameters with Guarantees

“…A theoretical study was proposed by [2], but the strategies in this paper are not feasible in practice. This problem was improved recently [11,5,12,17,36,16,14,21]. The closest work to this paper is [13], where the authors propose an efficient strategy to learn the starting point of online gradient descent.…”

Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

“…This thesis focuses on the first MAML algorithms, but the techniques here can be extended to analyze the Hessian-free multi-step MAML. Alternatively to meta-initialization algorithms such as MAML, meta-regularization approaches aim to learn a good bias for a regularized empirical risk minimization problem for intra-task learning [2,22,21,20,104,8,132]. [8] formalized a connection between meta-initialization and meta-regularization from an online learning perspective.…”