High Dimensional Bayesian Optimization Assisted by Principal Component Analysis

Abstract

Bayesian Optimization (BO) is a surrogate-assisted global optimization technique that has been successfully applied in various fields, e.g., automated machine learning and design optimization. Built upon a so-called infill-criterion and Gaussian Process regression (GPR), the BO technique suffers from a substantial computational complexity and hampered convergence rate as the dimension of the search spaces increases. Scaling up BO for high-dimensional optimization problems remains a challenging task. In this paper, we propose to tackle the scalability of BO by hybridizing it with a Principal Component Analysis (PCA), resulting in a novel PCA-assisted BO (PCA-BO) algorithm. Specifically, the PCA procedure learns a linear transformation from all the evaluated points during the run and selects dimensions in the transformed space according to the variability of evaluated points. We then construct the GPR model, and the infill-criterion in the space spanned by the selected dimensions. We assess the performance of our PCA-BO in terms of the empirical convergence rate and CPU time on multi-modal problems from the COCO benchmark framework. The experimental results show that PCA-BO can effectively reduce the CPU time incurred on high-dimensional problems, and maintains the convergence rate on problems with an adequate global structure. PCA-BO therefore provides a satisfactory trade-off between the convergence rate and computational efficiency opening new ways to benefit from the strength of BO approaches in high dimensional numerical optimization.

High-Dimensional Bayesian Optimisation with Large-Scale Constraints - An Application to Aeroelastic Tailoring

“…Additionally, there is a notable computational expense during hyperparameter tuning of the surrogate in the high-dimensional case. To mitigate this challenge, employing methods such as REMBO (Wang et al 2016) and ALEBO (Letham et al 2020) or (k)PCA-BO (Raponi et al 2020;Antonov et al 2022) presents an avenue for further reducing the computational cost. These methods operate under the assumption that certain dimensions are more significant than others, consequently reducing the number of tunable hyperparameters.…”

“…In Wang et al (2016), the authors use random projection methods to project the high-dimensional inputs to a lower dimensional subspace, ending up by constructing the GP model directly on the lower dimensional space, drastically reducing the number of hyperparameters. Raponi et al (2020) and Antonov et al (2022) use (kernel) Principal Component Analysis on the input space to identify a reduced set of dimensions based on the evaluated samples. Afterwards, the surrogate model is trained in this reduced dimensional space.…”

Human-in-the-Loop Robot Planning with Non-Contextual Bandit Feedback

“…However, these methods assume knowledge of the objective function, which is not available in the case of human satisfaction. On the other hand, the approaches in [16,17] project the original solution space into a low dimensional space using PCA or a variational autoencoder. Since the dimension reduction in these methods does not utilize objective function information, the low dimensional space may not contain the optimal solution and, therefore, it can introduce bias.…”