HEBO: An Empirical Study of Assumptions in Bayesian Optimisation

Abstract

In this work we rigorously analyse assumptions inherent to black-box optimisation hyper-parameter tuning tasks. Our results on the Bayesmark benchmark indicate that heteroscedasticity and non-stationarity pose significant challenges for black-box optimisers. Based on these findings, we propose a Heteroscedastic and Evolutionary Bayesian Optimisation solver (HEBO). HEBO performs non-linear input and output warping, admits exact marginal log-likelihood optimisation and is robust to the values of learned parameters. We demonstrate HEBO’s empirical efficacy on the NeurIPS 2020 Black-Box Optimisation challenge, where HEBO placed first. Upon further analysis, we observe that HEBO significantly outperforms existing black-box optimisers on 108 machine learning hyperparameter tuning tasks comprising the Bayesmark benchmark. Our findings indicate that the majority of hyper-parameter tuning tasks exhibit heteroscedasticity and non-stationarity, multiobjective acquisition ensembles with Pareto front solutions improve queried configurations, and robust acquisition maximisers afford empirical advantages relative to their non-robust counterparts. We hope these findings may serve as guiding principles for practitioners of Bayesian optimisation.

Effect of non-resonant background on the extraction of Raman signals from CARS spectra using deep neural networks

“…Moreover, the DL model hyper-parameters such as activation function, number of neurons, and number of layers can be modified to improve the performance. 53 Also, fine-tuning or transferring learning mechanisms can be explored to circumvent these limitations, which positively impacts model performance. It is also interesting to explore Gaussian processes 54 as an alternative model to extract Raman data from the CARS spectrum in future studies.…”

Predicting Visit Cost of Obstructive Sleep Apnea using Electronic Healthcare Records with Transformer

“…In one example, one might implement mortality/survival evaluation and cost prediction jointly in such a way that not only the financial element but also OSA patients' quality of life would be considered. Since abundant research has shown the power of Bayesian optimization for improved modeling, its use should be considered for informing prediction projects, although the method may prove time-consuming as the models grow larger and more complicated [78].…”

Data-driven discovery of molecular photoswitches with multioutput Gaussian processes

“…The two terms in the expression for the marginal likelihood represent the Occam factor 68 in their preference for selecting models of intermediate capacity. In practical applications, GPs have been primarily employed for their high quality uncertainty estimates across applications including materials modelling, 69 astronomical time series modelling, 70 machine learning hyperparameter tuning, 71,72 and Bayesian optimisation. 35,73…”