On the choice of the low-dimensional domain for global optimization via random embeddings

Abstract

The challenge of taking many variables into account in optimization problems may be overcome under the hypothesis of low effective dimensionality. Then, the search of solutions can be reduced to the random embedding of a low dimensional space into the original one, resulting in a more manageable optimization problem. Specifically, in the case of time consuming black-box functions and when the budget of evaluations is severely limited, global optimization with random embeddings appears as a sound alternative to random search. Yet, in the case of box constraints on the native variables, defining suitable bounds on a low dimensional domain appears to be complex. Indeed, a small search domain does not guarantee to find a solution even under restrictive hypotheses about the function, while a larger one may slow down convergence dramatically. Here we tackle the issue of low-dimensional domain selection based on a detailed study of the properties of the random embedding, giving insight on the aforementioned difficulties. In particular, we describe a minimal low-dimensional set in correspondence with the embedded search space. We additionally show that an alternative equivalent embedding procedure yields simultaneously a simpler definition of the low-dimensional minimal set and better properties in practice. Finally, the performance and robustness gains of the proposed enhancements for Bayesian optimization are illustrated on numerical examples.

Modeling and optimization with Gaussian processes in reduced eigenbases

“…In this catenoid example, the additive GP and the GP in the space of (all) 7 principal components achieve comparable results, both in terms of best value, and of function evaluations to attain the targets. Indeed, the true dimension (7) is relatively low, and we have noticed that the 5, 6 or even 7 first eigenshapes often got classified as active for the additive GP.…”

“…These results indicate a better performance of AddGP(α α α a + α α α a ) which benefits from the prioritization of the most influential eigenshapes in the additive model and, at the same time, accounts for all the 7 eigenshapes. Modeling in the space of the full α α α's (GP(α α α 1:7 )) performs fairly well too because the low true dimensionality (7). Despite its lower dimensionality, GP(α α α 1:4 ) does not work well.…”

“…In REMBO, a lower dimensional vector y ∈ R δ is embedded in X through a linear random embedding, y → A R y, where A R ∈ R D×δ is a random matrix. Instead of choosing a completely random and linear embedding with user-chosen (investigated in [7]) effective dimension , δ, our embedding is nonlinear (effect of the mapping φ(·)), supervised and semi-random (choice of the active/inactive directions). The dimension is no longer arbitrarily chosen since it is determined by the number of selected active components (Section 3.2.1), and the random part of the embedding is only associated to the inactive parts of α α α: denoting α α α a = (α a1 , .…”

Scaling Bayesian Optimization With Game Theory

“…The embedding dimension is specified a priori, i.e., no mechanism is provided to learn the embedding dimension. Recent work has investigated approaches to automate the selection of the embedding dimension [41].…”

Good practices for Bayesian optimization of high dimensional structured spaces

“…Wang et al 12 project the input producing the maximum of the acquisition function to the closest point on the boundary of the search space if it is outside it. Binois et al 26 and Binois et al 27 use projections to non-closest point to reduce the over-exploration of the boundaries of the search space.…”