Online Learning Rate Adaptation with Hypergradient Descent

Baydin, Atılım Güneş; Cornish, Robert; Rubio, David Martinez; Schmidt, Mark; Wood, Frank

doi:10.48550/arxiv.1703.04782

Cited by 14 publications

(20 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MLDG [30] introduces MAML to domain generalization by modifying the objective function. Alpha MAML [31] combines MAML with an online hyperparameter adaptation approach [32], thus avoiding tuning the learning rates. MAML++ [33] modifies MAML in several aspects and improves its stability and performances.…”

Section: B Gradient Based Meta-learningmentioning

confidence: 99%

Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?

Deng

Zhang

2021

2020 25th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

Substantial efforts have been made on improving the generalization abilities of deep neural networks (DNNs) in order to obtain better performances without introducing more parameters. On the other hand, meta-learning approaches exhibit powerful generalization on new tasks in few-shot learning. Intuitively, few-shot learning is more challenging than the standard supervised learning as each target class only has a very few or no training samples. The natural question that arises is whether the meta-learning idea can be used for improving the generalization of DNNs on the standard supervised learning. In this paper, we propose a novel meta-learning based training procedure (MLTP) for DNNs and demonstrate that the meta-learning idea can indeed improve the generalization abilities of DNNs. MLTP simulates the meta-training process by considering a batch of training samples as a task. The key idea is that the gradient descent step for improving the current task performance should also improve a new task performance, which is ignored by the current standard procedure for training neural networks. MLTP also benefits from all the existing training techniques such as dropout, weight decay, and batch normalization. We evaluate MLTP by training a variety of small and large neural networks on three benchmark datasets, i.e., CIFAR-10, CIFAR-100, and Tiny ImageNet. The experimental results show a consistently improved generalization performance on all the DNNs with different sizes, which verifies the promise of MLTP and demonstrates that the meta-learning idea is indeed able to improve the generalization of DNNs on the standard supervised learning.

show abstract

Section: B Gradient Based Meta-learningmentioning

confidence: 99%

Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?

Deng

Zhang

2021

2020 25th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

show abstract

“…We thus do not store network weights at multiple time steps, so gradient-based HPO becomes possible on previously-intractable large-scale problems. In essence, we develop an approximation to online hypergradient descent (Baydin et al, 2018).…”

Section: Hyperparameter Updatesmentioning

confidence: 99%

“…We demonstrate our algorithm handles a range of hyperparameter initialisations and datasets, improving test loss after a single training episode ('one pass'). Relaxing differentiation-through-optimisation (Domke, 2012) and hypergradient descent's (Baydin et al, 2018) exactness allows us to improve computational and memory efficiency. Our scalable one-pass method improves performance from arbitrary hyperparameter initialisations, and could be augmented with a further search over those initialisations if desired.…”

Section: Introductionmentioning

confidence: 99%

Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation

M.¹,

Oldewage²,

Hernández-Lobato³

2021

Preprint

View full text Add to dashboard Cite

Machine learning training methods depend plentifully and intricately on hyperparameters, motivating automated strategies for their optimisation. Many existing algorithms restart training for each new hyperparameter choice, at considerable computational cost. Some hypergradient-based one-pass methods exist, but these either cannot be applied to arbitrary optimiser hyperparameters (such as learning rates and momenta) or take several times longer to train than their base models. We extend these existing methods to develop an approximate hypergradient-based hyperparameter optimiser which is applicable to any continuous hyperparameter appearing in a differentiable model weight update, yet requires only one training episode, with no restarts. We also provide a motivating argument for convergence to the true hypergradient, and perform tractable gradient-based optimisation of independent learning rates for each model parameter. Our method performs competitively from varied random hyperparameter initialisations on several UCI datasets and Fashion-MNIST (using a one-layer MLP), Penn Treebank (using an LSTM) and CIFAR-10 (using a ResNet-18), in time only 2-3x greater than vanilla training.

show abstract

“…One interesting direction is to simultaneously optimize configurations and parameters, such methods have been recently explored in NAS [46], [201] and automated searching step-size for SGD [202], which have shown to be much more efficient than previous state-of-the-arts.…”

Section: Techniquesmentioning

confidence: 99%

Taking Human out of Learning Applications: A Survey on Automated Machine Learning

Yao,

Wang,

Chen

et al. 2018

Preprint

102

110

View full text Add to dashboard Cite

Machine learning techniques have deeply rooted in our everyday life. However, since it is knowledge-and labor-intensive to pursue good learning performance, humans are heavily involved in every aspect of machine learning. To make machine learning techniques easier to apply and reduce the demand for experienced human experts, automated machine learning (AutoML) has emerged as a hot topic with both industrial and academic interest. In this paper, we provide an up to date survey on AutoML. First, we introduce and define the AutoML problem, with inspiration from both realms of automation and machine learning. Then, we propose a general AutoML framework that not only covers most existing approaches to date, but also can guide the design for new methods. Subsequently, we categorize and review the existing works from two aspects, i.e., the problem setup and the employed techniques. The proposed framework and taxonomies provide a detailed analysis of AutoML approaches and explain the reasons underneath their successful applications. We hope this survey can serve as not only an insightful guideline for AutoML beginners but also an inspiration for future research.

show abstract

Online Learning Rate Adaptation with Hypergradient Descent

Cited by 14 publications

References 0 publications

Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?

Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?

Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation

Taking Human out of Learning Applications: A Survey on Automated Machine Learning

Contact Info

Product

Resources

About