Confidence Bound Minimization for Bayesian optimization with Student's-t Processes

Abstract

Bayesian optimization seeks the global optimum of a black-box, objective function f (x), in the fewest possible iterations. Recent work applied knowledge of the true value of the optimum to the Gaussian Process probabilistic model typically used in Bayesian optimization. This, together with a new acquisition function called Confidence Bound Minimization, resulted in a Gaussian probabilistic posterior in which the predictions were no greater than the known maximum (and no less than for minimum). Our novel work applies Confidence Bound Minimization to Bayesian optimization with Student's-t Processes, a probabilistic alternative which addresses known weaknesses in Gaussian Processes-outliers' probability and the calculation of posterior covariance. The new model is applied to the problem of hyperparameter tuning for an XGBoost classifier. Experiments show superior regret minimization and predictive accuracy, versus the popular Expected Improvement acquisition function. Combining Confidence Bound Minimization with a transformed Student's-t Process probabilistic model and known optima produces superior training regret minimization and posterior predictions for the Six-Hump Camel(2D) and Levy(4D) benchmark problems, which do not fall below true minima.

Expected Regret Minimization for Bayesian Optimization with Student's-t Processes

“…Our work enhances Bayesian optimization by comparing STP ( = 5) ERM versus GP ERM [9]. The data is split 85/15 between training and testing [18]. 3-fold cross-validation of the XGBoost classifier is averaged to measure [9,18].…”

“…The data is split 85/15 between training and testing [18]. 3-fold cross-validation of the XGBoost classifier is averaged to measure [9,18]. We use a logistic objective function [9,18], with 5 random initializations [19] and = 30 post-initialization iterations [9,18].…”

“…3-fold cross-validation of the XGBoost classifier is averaged to measure [9,18]. We use a logistic objective function [9,18], with 5 random initializations [19] and = 30 post-initialization iterations [9,18]. The surrogacy training results are shown in Figure 5 and independently repeated 20 times [9] for both STP ( = 5) ERM and GP ERM.…”