Gradient-based Hyperparameter Optimization Over Long Horizons

Abstract

Gradient-based hyperparameter optimization is an attractive way to perform metalearning across a distribution of tasks, or improve the performance of an optimizer on a single task. However, this approach has been unpopular for tasks requiring long horizons (many gradient steps), due to memory scaling and gradient degradation issues. A common workaround is to learn hyperparameters online or split the horizon into smaller chunks. However, this introduces greediness which comes with a large performance drop, since the best local hyperparameters can make for poor global solutions. In this work, we enable non-greediness over long horizons with a two-fold solution. First, we share hyperparameters that are contiguous in time, and show that this drastically mitigates gradient degradation issues. Then, we derive a forward-mode differentiation algorithm for the popular momentumbased SGD optimizer, which allows for a memory cost that is constant with horizon size. When put together, these solutions allow us to learn hyperparameters without any prior knowledge. Compared to the baseline of hand-tuned off-theshelf hyperparameters, our method compares favorably on simple datasets like SVHN. On CIFAR-10 we match the baseline performance, and demonstrate for the first time that learning rate, momentum and weight decay schedules can be learned with gradients on a dataset of this size. Code is available at: https: //github.com/polo5/NonGreedyGradientHPO Preprint. Under review.

Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation

“…As future work, our myopic approach could be extended to longer horizons by incorporating the principles of recent work by Micaelli & Storkey (2020), which presents a promising research direction. We also depend inconveniently on meta-hyperparameters, which are not substantially tuned.…”

“…Modern developments include theoretical reformulations of bilevel optimisation to improve performance (Liu et al, 2020;Li et al, 2020), optimising distinct hyperparameters for each model parameter (Lorraine et al, 2019;Jie et al, 2020), and computing forward-mode hypergradient averages using more exact techniques than we do (Micaelli & Storkey, 2020). Although these approaches increase computational efficiency and the range of tunable parameters, achieving both benefits at once remains challenging.…”

Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies (Extended Abstract)

“…There have also been attempts to use forward-mode gradient accumulation for hyperparameter optimization (Franceschi et al, 2017), which is only tractable when the hyperparameter dimensionality is very small (e.g., < 10). Most gradient-based approaches perform online, joint optimization over the model parameters and hyperparameters; a notable exception is Micaelli & Storkey (2020), that performs offline updates after each full inner optimization run. Black-box approaches typically do not scale well beyond ∼ 10 hyperparameters.…”

Hyper-Learning for Gradient-Based Batch Size Adaptation

“…Gradient-based methods were developed to access this structure, making them scalable to tune millions of hyperparameters within deep architectures [10,29]. Gradients have been implemented extensively within HO, including to learning rates [9,30,31], regularization coefficients [10,32,11], and neural architecture search [33]. Another class of algorithms view the HO problem as a gray-box by considering the inner optimization structure, but relax the need for the meta-objective to be differentiable: population-based training [34], hypernetwork-based HO [12,35], and persistent evolution strategies (PES) [13] are examples of gray-box approaches.…”