DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization

Awad, Noor H.; Mallik, Neeratyoy; Hutter, Frank

doi:10.24963/ijcai.2021/296

Cited by 26 publications

(30 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As mentioned earlier, a major source of sub-optimality for the current gray-box techniques (Li et al, 2017;Falkner et al, 2018;Awad et al, 2021) is the poor rank correlation of performances at low and high budgets, which is endemic to the successive halving mechanism. In essence, prior methods mistakenly discard good hyper-parameter configurations by myopically relying only on the early performance after a few epochs, following a fixed fidelity scheme.…”

Section: Motivationmentioning

confidence: 99%

“…BOHB (Falkner et al, 2018) uses TPE (Bergstra et al, 2011) and builds a surrogate model for every fidelity adhering to a fixed-fidelity selection scheme. DEHB (Awad et al, 2021) samples candidates using differential evolution which handles discrete and large hyperparameter search spaces better than BOHB.…”

Section: Related Work On Gray-box Hpomentioning

confidence: 99%

“…BOHB builds a model for every fidelity that is considered. DEHB: A method that builds upon Hyperband by exploiting differential evolution to sample the initial candidates of a Hyperband bracket (Awad et al, 2021).…”

Section: Baselinesmentioning

confidence: 99%

“…different number of epochs) and successively halves these configurations by keeping only the top performers. Follow-up works, such as BOHB (Falkner et al, 2018) or DEHB (Awad et al, 2021), replace the random sampling of Hyperband with sampling based on Bayesian optimization or differentiable evolution.…”

Section: Introductionmentioning

confidence: 99%

“…The joint effect of modeling a GP kernel across budgets together with a dedicated acquisition function leads to DYHPO achieving a statistically significant empirical gain against state-of-the-art gray-box baselines (Falkner et al, 2018;Awad et al, 2021), including prior work on multibudget GPs (Kandasamy et al, 2017; or neural networks (Li et al, 2020b). We demonstrate the performance of DYHPO in three diverse popular types of deep learning architectures (MLP, CNN/NAS, RNN) and 50 datasets of three diverse modalities (tabular, image, natural language processing).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning

Wistuba¹,

Kadra²,

Grabocka³

2022

Preprint

View full text Add to dashboard Cite

Gray-box hyperparameter optimization techniques have recently emerged as a promising direction for tuning Deep Learning methods. In this work, we introduce DyHPO, a method that learns to dynamically decide which configuration to try next, and for what budget. Our technique is a modification to the classical Bayesian optimization for a gray-box setup. Concretely, we propose a new surrogate for Gaussian Processes that embeds the learning curve dynamics and a new acquisition function that incorporates multi-budget information. We demonstrate the significant superiority of DyHPO against stateof-the-art hyperparameter optimization baselines through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse neural networks (MLP, CNN/NAS, RNN).

show abstract

Section: Motivationmentioning

confidence: 99%

Section: Related Work On Gray-box Hpomentioning

confidence: 99%

Section: Baselinesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning

Wistuba¹,

Kadra²,

Grabocka³

2022

Preprint

View full text Add to dashboard Cite

show abstract

Ranking‐based architecture generation for surrogate‐assisted neural architecture search

Xiao,

Wang

2024

Concurrency and Computation

View full text Add to dashboard Cite

Architectures generation optimization has been received a lot of attention in neural architecture search (NAS) since its efficiency in generating architecture. By learning the architecture representation through unsupervised learning and constructing a latent space, the prediction process of predictors is simplified, leading to improved efficiency in architecture search. However, searching for architectures with top performance in complex and large NAS search spaces remains challenging. In this paper, an approach that combined a ranker and generative model is proposed to address this challenge through regularizing the latent space and identifying architectures with top rankings. We introduce the ranking error to gradually regulate the training of the generative model, making it easier to identify architecture representations in the latent space. Additionally, a surrogate‐assisted evolutionary search method that utilized neural network Bayesian optimization is proposed to efficiently explore promising architectures in the latent space. We demonstrate the benefits of our approach in optimizing architectures with top rankings, and our method outperforms state‐of‐the‐art techniques on various NAS benchmarks. The code is available at https://github.com/outofstyle/RAGS‐NAS.

show abstract

Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges

Bischl

Binder

Lang

et al. 2023

WIREs Data Min & Knowl

222

View full text Add to dashboard Cite

Most machine learning algorithms are configured by a set of hyperparameters whose values must be carefully chosen and which often considerably impact performance. To avoid a time-consuming and irreproducible manual process of trial-anderror to find well-performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods-for example, based on resampling error estimation for supervised machine learning-can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods, from simple techniques such as grid or random search to more advanced methods like evolution strategies, Bayesian optimization, Hyperband, and racing. This work gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with machine learning pipelines, runtime improvements, and parallelization.

show abstract

DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization

Cited by 26 publications

References 15 publications

Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning

Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning

Ranking‐based architecture generation for surrogate‐assisted neural architecture search

Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges

Contact Info

Product

Resources

About