COmparing Continuous Optimizers: numbbo/COCO on Github

Hansen, Nikolaus; Brockhoff, Dimo; Mersmann, Olaf; Tušar, Tea; Tušar, Dejan; Elhara, Ouassim Ait; Sampaio, Phillipe R.; Atamna, Asma; Varelas, Konstantinos; Batu, Umut; Nguyen, Duc Manh; Matzner, Filip; Auger, Anne

doi:10.5281/zenodo.2594848

Cited by 20 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An instance is a rotated or shifted version of the original objective function. All described experiments were run with a recent GitHub version 2 , v2.3.1 [15]. Each algorithm is tested on the 15 available standard instances of the BBOB function set.…”

Section: Test Functionsmentioning

confidence: 99%

Expected Improvement versus Predicted Value in Surrogate-Based Optimization

Rehbach¹,

Zaefferer²,

Naujoks³

et al. 2020

Preprint

View full text Add to dashboard Cite

Surrogate-based optimization relies on so-called infill criteria (acquisition functions) to decide which point to evaluate next. When Kriging is used as the surrogate model of choice (also called Bayesian optimization), one of the most frequently chosen criteria is expected improvement. We argue that the popularity of expected improvement largely relies on its theoretical properties rather than empirically validated performance. Few results from the literature show evidence, that under certain conditions, expected improvement may perform worse than something as simple as the predicted value of the surrogate model. We benchmark both infill criteria in an extensive empirical study on the 'BBOB' function set. This investigation includes a detailed study of the impact of problem dimensionality on algorithm performance. The results support the hypothesis that exploration loses importance with increasing problem dimensionality. A statistical analysis reveals that the purely exploitative search with the predicted value criterion performs better on most problems of five or higher dimensions. Possible reasons for these results are discussed. In addition, we give an in-depth guide for choosing the infill criteria based on prior knowledge about the problem at hand, its dimensionality, and the available budget.

show abstract

Section: Test Functionsmentioning

confidence: 99%

Expected Improvement versus Predicted Value in Surrogate-Based Optimization

Rehbach¹,

Zaefferer²,

Naujoks³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Experimental Setup. We assess the performance of PCA-BO on ten multi-modal functions taken from the the BBOB problem set [11], and compare the experimental result to a standard BO. This choice of test functions aims to verify the applicability of PCA-BO to more difficult problems, which potentially resemble the feature of real-world applications.…”

Section: Methodsmentioning

confidence: 99%

“…Here, the weighting scheme is meant for taking the objective values into account. We assessed the empirical performance of PCA-BO by testing it on the well-known BBOB problem set [11]. Among these problems, we focus on the multi-modal ones, which are most representative of real-world optimization challenges.…”

Section: Introductionmentioning

confidence: 99%

High Dimensional Bayesian Optimization Assisted by Principal Component Analysis

Raponi¹,

Wang²,

Bujny³

et al. 2020

Preprint

View full text Add to dashboard Cite

Bayesian Optimization (BO) is a surrogate-assisted global optimization technique that has been successfully applied in various fields, e.g., automated machine learning and design optimization. Built upon a so-called infill-criterion and Gaussian Process regression (GPR), the BO technique suffers from a substantial computational complexity and hampered convergence rate as the dimension of the search spaces increases. Scaling up BO for high-dimensional optimization problems remains a challenging task. In this paper, we propose to tackle the scalability of BO by hybridizing it with a Principal Component Analysis (PCA), resulting in a novel PCA-assisted BO (PCA-BO) algorithm. Specifically, the PCA procedure learns a linear transformation from all the evaluated points during the run and selects dimensions in the transformed space according to the variability of evaluated points. We then construct the GPR model, and the infill-criterion in the space spanned by the selected dimensions. We assess the performance of our PCA-BO in terms of the empirical convergence rate and CPU time on multi-modal problems from the COCO benchmark framework. The experimental results show that PCA-BO can effectively reduce the CPU time incurred on high-dimensional problems, and maintains the convergence rate on problems with an adequate global structure. PCA-BO therefore provides a satisfactory trade-off between the convergence rate and computational efficiency opening new ways to benefit from the strength of BO approaches in high dimensional numerical optimization.

show abstract

“…The objective is generated by sampling a c uniformly at random on the hyper-sphere. Following the idea in Hansen et al [2016Hansen et al [ , 2019, we define the constraints of such problems by setting the gradient of the first constraint to a 1 = −c to ensure the Karush-Kuhn-Tucker optimality conditions Kuhn and Tucker [1951], Nocedal and Wright [2006] hold at (0, . .…”

Section: Appendix a Convex Optimization Numerical Illustrationmentioning

confidence: 99%

Convex Optimization with an Interpolation-based Projection and its Application to Deep Learning

Akrour¹,

Atamna²,

Peters³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Convex optimizers have known many applications as differentiable layers within deep neural architectures. One application of these convex layers is to project points into a convex set. However, both forward and backward passes of these convex layers are significantly more expensive to compute than those of a typical neural network. We investigate in this paper whether an inexact, but cheaper projection, can drive a descent algorithm to an optimum. Specifically, we propose an interpolation-based projection that is computationally cheap and easy to compute given a convex, domain defining, function. We then propose an optimization algorithm that follows the gradient of the composition of the objective and the projection and prove its convergence for linear objectives and arbitrary convex and Lipschitz domain defining inequality constraints. In addition to the theoretical contributions, we demonstrate empirically the practical interest of the interpolation projection when used in conjunction with neural networks in a reinforcement learning and a supervised learning setting.

show abstract

COmparing Continuous Optimizers: numbbo/COCO on Github

Cited by 20 publications

References 0 publications

Expected Improvement versus Predicted Value in Surrogate-Based Optimization

Expected Improvement versus Predicted Value in Surrogate-Based Optimization

High Dimensional Bayesian Optimization Assisted by Principal Component Analysis

Convex Optimization with an Interpolation-based Projection and its Application to Deep Learning

Contact Info

Product

Resources

About