On the role of Model Uncertainties in Bayesian Optimization

Abstract

Bayesian optimization (BO) is a popular method for black-box optimization, which relies on uncertainty as part of its decision-making process when deciding which experiment to perform next. However, not much work has addressed the effect of uncertainty on the performance of the BO algorithm and to what extent calibrated uncertainties improve the ability to find the global optimum. In this work, we provide an extensive study of the relationship between the BO performance (regret) and uncertainty calibration for popular surrogate models and compare them across both synthetic and real-world experiments. Our results confirm that Gaussian Processes are strong surrogate models and that they tend to outperform other popular models. Our results further show a positive association between calibration error and regret, but interestingly, this association disappears when we control for the type of model in the analysis. We also studied the effect of re-calibration and demonstrate that it generally does not lead to improved regret. Finally, we provide theoretical justification for why uncertainty calibration might be difficult to combine with BO due to the small sample sizes commonly used.Uncertainty also plays a central role Bayesian Optimization (BO) (Snoek et al., 2012), which will be the focus of this paper. As a sequential design strategy for global optimization, BO has several applications with perhaps the most popular ones being general experimental design (Shahriari et al., 2015) and model selection for machine learning

Calibration of uncertainty in the active learning of machine learning force fields

“…This is problematic because, as previously stated, data generation for FFLUX is computationally expensive. Without sufficient additional data, post-hoc calibration may have limited effect in active learning, as has been shown to be the case in the related field of Bayesian optimization [25]. This motivates the pursuit of a model that provides reliable uncertainty estimates without the need for an additional calibration set.…”