Bayesian Optimization for Categorical and Category-Specific Continuous Inputs

Abstract

Many real-world functions are defined over both categorical and category-specific continuous variables and thus cannot be optimized by traditional Bayesian optimization (BO) methods. To optimize such functions, we propose a new method that formulates the problem as a multi-armed bandit problem, wherein each category corresponds to an arm with its reward distribution centered around the optimum of the objective function in continuous variables. Our goal is to identify the best arm and the maximizer of the corresponding continuous function simultaneously. Our algorithm uses a Thompson sampling scheme that helps connecting both multi-arm bandit and BO in a unified framework. We extend our method to batch BO to allow parallel optimization when multiple resources are available. We theoretically analyze our method for convergence and prove sub-linear regret bounds. We perform a variety of experiments: optimization of several benchmark functions, hyper-parameter tuning of a neural network, and automatic selection of the best machine learning model along with its optimal hyper-parameters (a.k.a automated machine learning). Comparisons with other methods demonstrate the effectiveness of our proposed method.

Black-box mixed-variable optimisation using a surrogate model that satisfies integer constraints

“…It makes use of a mix of multi-armed bandits and Gaussian processes. Similar ideas are utilised in [22,28]. Another interesting new research direction is to combine the advantages of Gaussian processes and artificial neural networks [17], although more research is required to make this computationally feasible for larger problems.…”

Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces

“…Linear models [3], Gaussian processes with kernels over discrete structures [46,15,24,36], and random forest [30] are considered for surrogate modeling. Search strategies include mathematical optimization [3,13,15], heuristic search methods [30,46,14], and combination thereof [16,66,44]. [31] provided a BO approach for tree structured spaces.…”

Explaining Inference Queries with Bayesian Optimization

“…The key technical challenge is that existing BO approaches [9,20,42] cannot be naively adapted to explanation generation. In the hyperparameter tuning setting, categorical variables typically have very low cardinality (e.g., with 2-3 distinct values [30]). In the query explanation setting, however, a categorical variable can have many more distinct values.…”

“…However, it does not scale well to variables with many distinct values [35]. BO may use tree-based surrogate models (e.g., random forests [20], tree Parzen estimators [9]) to handle categorical variables, however their predictive accuracy is empirically poor [16,30]. Other work optimizes a combinatorial search space [5,12,31], and categorical/category-specific continuous variables [30].…”