HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation

Cowen-Rivers, Alexander I.; Lyu, Wenlong; Tutunov, Rasul; Wang, Zhi; Grosnit, Antoine; Griffiths, Ryan Rhys; Maraval, Alexandre Max; Hao, Jianye; Wang, Jun; Peters, Jan; Ammar, Haitham Bou

doi:10.48550/arxiv.2012.03826

Cited by 8 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Acquisition functions of BO: HEBO uses a multi-objective acquisition ensemble that aims to find Pareto-optimal points [14]. DEEP-BO simply uses a round-robin of BO models where EI, PI, and UCB take turns.…”

Section: Discussionmentioning

confidence: 99%

“…Input and output transformations: HEBO is the winner of the black-box optimization (BBO) challenge at NeurIPS 2020 [61], where the challenge focused on evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models. To deal with the complexities associated with the competition datasets, both input and output nonlinear transformations were used [14]. More specifically, they utilized an input warped GP [60] to handle non-stationary functions and output transformations [56] to model non-Gaussian data.…”

Section: Related Workmentioning

confidence: 99%

“…The GP model was based on the InputWarpedGP class of the GPy library [27] and our RF model was based on the RandomForestRegressor class of the scikit-learn library [52], which is open-sourced under a BSD license. More specifically, the kernel design in our GP model is almost the same as HEBO [14], where the linear and Mat'ern 3/2 kernels were used with automatic relevance determination. However, its hyper-parameters were solely sampled using Monte Carlo estimation of the acquisition function values to reduce the modeling cost of GP regression.…”

Section: B Implementation Detailsmentioning

confidence: 99%

“…• Output transformation: To stabilize variance and minimize skewness in the power transformation, the box-cox and yeo-johnson transformations were employed in the same way as in HEBO [14]. For each transformation, the power transform implementation from the scikit-learn was used.…”

Section: B Implementation Detailsmentioning

confidence: 99%

See 3 more Smart Citations

B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization Modules for Neural Architecture Search

Cho¹,

Shin²,

Rhee³

2022

Preprint

View full text Add to dashboard Cite

The early pioneering Neural Architecture Search (NAS) works were multi-trial methods applicable to any general search space. The subsequent works took advantage of the early findings and developed weight-sharing methods that assume a structured search space typically with pre-fixed hyperparameters. Despite the amazing computational efficiency of the weight-sharing NAS algorithms, it is becoming apparent that multi-trial NAS algorithms are also needed for identifying very high-performance architectures, especially when exploring a general search space. In this work, we carefully review the latest multi-trial NAS algorithms and identify the key strategies including Evolutionary Algorithm (EA), Bayesian Optimization (BO), diversification, input and output transformations, and lower fidelity estimation. To accommodate the key strategies into a single framework, we develop B 2 EA that is a surrogate assisted EA with two BO surrogate models and a mutation step in between. To show that B 2 EA is robust and efficient, we evaluate three performance metrics over 14 benchmarks with general and cellbased search spaces. Comparisons with state-of-the-art multi-trial algorithms reveal that B 2 EA is robust and efficient over the 14 benchmarks for three difficulty levels of target performance. The B 2 EA code is publicly available at https: //github.com/snu-adsl/BBEA.Preprint. Under review.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: B Implementation Detailsmentioning

confidence: 99%

Section: B Implementation Detailsmentioning

confidence: 99%

See 2 more Smart Citations

B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization Modules for Neural Architecture Search

Cho¹,

Shin²,

Rhee³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Note that the assumption 2 allows heteroscedasticity among different arms since L 2 can be chosen as the largest variance among arms. Such heteroscedasticity consideration arises and has been identified as a challenge in applications of Bayesian optimization (Kirschner, 2021;Cowen-Rivers et al, 2020).…”

Section: Stochastic Linear Bandit Problemmentioning

confidence: 99%

Residual Bootstrap Exploration for Stochastic Linear Bandit

Wu¹,

Wang²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose a new bootstrap-based online algorithm for stochastic linear bandit problems. The key idea is to adopt residual bootstrap exploration, in which the agent estimates the next step reward by re-sampling the residuals of mean reward estimate. Our algorithm, residual bootstrap exploration for stochastic linear bandit (LinReBoot), estimates the linear reward from its re-sampling distribution and pulls the arm with the highest reward estimate. In particular, we contribute a theoretical framework to demystify residual bootstrap-based exploration mechanisms in stochastic linear bandit problems. The key insight is that the strength of bootstrap exploration is based on collaborated optimism between the online-learned model and the re-sampling distribution of residuals. Such observation enables us to show that the proposed LinReBoot secure a high-probability O(d √ n) sub-linear regret under mild conditions. Our experiments support the easy generalizability of the ReBoot principle in the various formulations of linear bandit problems and show the significant computational efficiency of LinReBoot.

show abstract

System architecture optimization strategies: dealing with expensive hierarchical problems

Bussemaker,

Saves,

Bartoli

et al. 2024

J Glob Optim

View full text Add to dashboard Cite

Choosing the right system architecture for the problem at hand is challenging due to the large design space and high uncertainty in the early stage of the design process. Formulating the architecting process as an optimization problem may mitigate some of these challenges. This work investigates strategies for solving system architecture optimization (SAO) problems: expensive, black-box, hierarchical, mixed-discrete, constrained, multi-objective problems that may be subject to hidden constraints. Imputation ratio, correction ratio, correction fraction, and max rate diversity metrics are defined for characterizing hierarchical design spaces. This work considers two classes of optimization algorithms for SAO: multi-objective evolutionary algorithms such as NSGA-II, and Bayesian optimization (BO) algorithms. A new Gaussian process kernel is presented that enables modeling hierarchical categorical variables, extending previous work on modeling continuous and integer hierarchical variables. Next, a hierarchical sampling algorithm that uses design space hierarchy to group design vectors by active design variables is developed. Then, it is demonstrated that integrating more hierarchy information in the optimization algorithms yields better optimization results for BO algorithms. Several realistic single-objective and multi-objective test problems are used for investigations. Finally, the BO algorithm is applied to a jet engine architecture optimization problem. This work shows that the developed BO algorithm can effectively solve the problem with one order of magnitude less function evaluations than NSGA-II. The algorithms and problems used in this work are implemented in the open-source Python library SBArchOpt.

show abstract

HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation

Cited by 8 publications

References 0 publications

B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization Modules for Neural Architecture Search

B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization Modules for Neural Architecture Search

Residual Bootstrap Exploration for Stochastic Linear Bandit

System architecture optimization strategies: dealing with expensive hierarchical problems

Contact Info

Product

Resources

About