IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics

Nobel, Jacob de; Ye, Furong; Vermetten, Diederick; Wang, Hao; Doerr, Benjamin; Bäck, Thomas

doi:10.48550/arxiv.2111.04077

Cited by 3 publications

(4 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To date, hundreds of different optimization algorithms have been tested on the BBOB suite [1]. Over the years, the BBOB suite has been integrated into other benchmarking toolsas a whole, as is the case for BBOBtorch [24] and IOHexperimenter [5], or in parts, as is the case for the Nevergrad platform [27].…”

Section: The Bbob Problem Suitementioning

confidence: 99%

See 1 more Smart Citation

Using Affine Combinations of BBOB Problems for Performance Assessment

Vermetten

Doerr

2023

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

Choosing a set of benchmark problems is often a key component of any empirical evaluation of iterative optimization heuristics. In continuous, single-objective optimization, several sets of problems have become widespread, including the well-established BBOB suite. While this suite is designed to enable rigorous benchmarking, it is also commonly used for testing methods such as algorithm selection, which the suite was never designed around.We present the MA-BBOB function generator, which uses the BBOB suite as component functions in an affine combination. In this work, we describe the full procedure to create these affine combinations and highlight the trade-offs of several design decisions, specifically the choice to place the optimum uniformly at random in the domain. We then illustrate how this generator can be used to gain more low-level insight into the function landscapes through the use of exploratory landscape analysis.Finally, we show a potential use-case of MA-BBOB in generating a wide set of training and testing data for algorithm selectors.Using this setup, we show that the basic scheme of using a set of landscape features to predict the best algorithm does not lead to optimal results, and that an algorithm selector trained purely on the BBOB functions generalizes poorly to the affine combinations.CCS Concepts: • Theory of computation → Random search heuristics; Design and analysis of algorithms; Bio-inspired optimization.

show abstract

Section: The Bbob Problem Suitementioning

confidence: 99%

“…The MA-BBOB generator as described here is made available directly in the IOHexperimenter package [5] as part of the IOHprofiler project [7]. This enables us to use any of the built-in logging and tracking options of IOHexperimenter.…”

Section: Availability and Reproducibilitymentioning

confidence: 99%

Using Affine Combinations of BBOB Problems for Performance Assessment

Vermetten

Doerr

2023

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

show abstract

“…The 𝐿 ∞ star discrepancy problem was included as a black-box benchmark problem in IOHexperimenter (version 0.3.7) [7], using three different point set generators:…”

Section: Numerical Black-box Optimization Approachesmentioning

confidence: 99%

“…(1) Diagonal Covariance Matrix Adaptation Evolution Strategy (dCMA-ES) [20] (2) NGOpt14, Nevergrad's algorithm selection wizard [26] (3) Estimation of Multivariate Normal Algorithm (EMNA) [25] (4) Differential Evolution [40] (5) Constrained Optimization BY Linear Approximation (Cobyla) [32] (6) Random Search (7) Particle Swarm Optimization (PSO) [22] (8) Simultaneous Perturbation Stochastic Approximation algorithm (SPSA) [39] The algorithms are chosen "as they are" from Nevergrad. That is, we did not perform any hyper-parameter tuning nor did we change any of their components.…”

Section: Numerical Black-box Optimization Approachesmentioning

confidence: 99%

Computing Star Discrepancies with Numerical Black-Box Optimization Algorithms

François¹,

Vermetten²,

Nobel³

et al. 2023

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

The 𝐿 ∞ star discrepancy is a measure for the regularity of a finite set of points taken from [0, 1) 𝑑 . Low discrepancy point sets are highly relevant for Quasi-Monte Carlo methods in numerical integration and several other applications. Unfortunately, computing the 𝐿 ∞ star discrepancy of a given point set is known to be a hard problem, with the best exact algorithms falling short for even moderate dimensions around 8. However, despite the difficulty of finding the global maximum that defines the 𝐿 ∞ star discrepancy of the set, local evaluations at selected points are inexpensive. This makes the problem tractable by black-box optimization approaches.In this work we compare 8 popular numerical black-box optimization algorithms on the 𝐿 ∞ star discrepancy computation problem, using a wide set of instances in dimensions 2 to 15. We show that all used optimizers perform very badly on a large majority of the instances and that in many cases random search outperforms even the more sophisticated solvers. We suspect that state-of-the-art numerical black-box optimization techniques fail to capture the global structure of the problem, an important shortcoming that may guide their future development.We also provide a parallel implementation of the best-known algorithm to compute the discrepancy. CCS CONCEPTS• Computing methodologies → Randomized search; Parallel algorithms.

show abstract