Per instance algorithm configuration of CMA-ES with limited budget

Belkhir, Nacim; Dréo, Johann; Savéant, Pierre; Schoenauer, Marc

doi:10.1145/3071178.3071343

Cited by 82 publications

(72 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such feature-based analyses are at the heart of algorithm selection techniques [Kerschke et al(2018) Kerschke, Hoos, Neumann, and Trautmann], which use landscape features and performance data to build a model that predicts how the tested algorithms will perform on a previously unseen problem. Similar approaches can be found in per-instance-algorithm configuration (PIAC) approaches, which have recently shown very promising performance in the context of continuous black-box optimiza-tion [Belkhir et al(2017)Belkhir, Dréo, Savéant, and Schoenauer]. A key step towards such a feature-based performance analysis are the selection and the efficient computation of meaningful features.…”

Section: Discussionmentioning

confidence: 92%

“…The Kolmogorov-Smirnov tests are available in IOHANALYZER, while extension, in particular in terms of nonparametric tests that are designed for two or more samples, e.g., Kruskal-Wallis or Friedman test, are left for future work. For the large-scale multiple testing scenario (thousands of pairwise tests are performed, for instance), we are considering to add Bayesian inference, as suggested, e.g., in [Benavoli et al(2017)Benavoli, Corani, Demsar, and Zaffalon]. Such an approach has recently been proposed for comparing performance of IOHs [Calvo et al(2018)Calvo, Ceberio, and Lozano] and has been applied to the data set of this paper in [Calvo et al(2019)Calvo, Shir, Ceberio, Doerr, Wang, Bäck, and Lozano].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Benchmarking discrete optimization heuristics with IOHprofiler

Doerr

Ye²,

Horesh³

et al. 2019

Proceedings of the Genetic and Evolutionary Computation Conference Companion

View full text Add to dashboard Cite

Automated benchmarking environments aim to support researchers in understanding how different algorithms perform on different types of optimization problems. Such comparisons provide insights into the strengths and weaknesses of different approaches, which can be leveraged into designing new algorithms and into the automation of algorithm selection and configuration. With the ultimate goal to create a meaningful benchmark set for iterative optimization heuristics, we have recently released IOHprofiler, a software built to create detailed performance comparisons between iterative optimization heuristics. With this present work we demonstrate that IOHprofiler provides a suitable environment for automated benchmarking. We compile and assess a selection of 23 discrete optimization problems that subscribe to different types of fitness landscapes. For each selected problem we compare performances of twelve different heuristics, which are as of now available as baseline algorithms in IOHprofiler. We also provide a new module for IOHprofiler which extents the fixed-target and fixed-budget results for the individual problems by ECDF results, which allows one to derive aggregated performance statistics for groups of problems.

show abstract

Section: Discussionmentioning

confidence: 92%

Section: Discussionmentioning

confidence: 99%

Benchmarking discrete optimization heuristics with IOHprofiler

Doerr

Ye²,

Horesh³

et al. 2019

Proceedings of the Genetic and Evolutionary Computation Conference Companion

View full text Add to dashboard Cite

show abstract

“…The per-instance variant of the algorithm configuration problem, which can be seen as a generalisation of per-instance algorithm selection, largely remains an open challenge (see, e.g., Hutter et al, 2006;Belkhir et al, 2016Belkhir et al, , 2017, and we briefly discuss it further in Section 6.…”

Section: Algorithm Selection and Related Problemsmentioning

confidence: 99%

“…Note that by using a platform-independent web-application of the flacco package 1 (Hanster and Kerschke, 2017), researchers and practitioners, who are unfamiliar with R, can also benefit from this extensive collection of more than 300 landscape features. Belkhir et al (2016Belkhir et al ( , 2017 were among the first to leverage the ELA features provided by flacco for per-instance algorithm configuration.…”

Section: Continuous Problemsmentioning

confidence: 99%

Automated Algorithm Selection: Survey and Perspectives

Kerschke

Hoos

Neumann

et al. 2019

Evolutionary Computation

317

184

View full text Add to dashboard Cite

It has long been observed that for practically any computational problem that has been intensely studied, different instances are best solved using different algorithms. This is particularly pronounced for computationally hard problems, where in most cases, no single algorithm defines the state of the art; instead, there is a set of algorithms with complementary strengths. This performance complementarity can be exploited in various ways, one of which is based on the idea of selecting, from a set of given algorithms, for each problem instance to be solved the one expected to perform best. The task of automatically selecting an algorithm from a given set is known as the per-instance algorithm selection problem and has been intensely studied over the past 15 years, leading to major improvements in the state of the art in solving a growing number of discrete combinatorial problems, including propositional satisfiability and AI planning. Perinstance algorithm selection also shows much promise for boosting performance in solving continuous and mixed discrete/continuous optimisation problems.This survey provides an overview of research in automated algorithm selection, ranging from early and seminal works to recent and promising application areas. Different from earlier work, it covers applications to discrete and continuous problems, and discusses algorithm selection in context with conceptually related approaches, such as algorithm configuration, scheduling or portfolio selection. Since informative and cheaply computable problem instance features provide the basis for effective perinstance algorithm selection systems, we also provide an overview of such features for discrete and continuous problems. Finally, we provide perspectives on future work in the area and discuss a number of open research challenges.

show abstract

“…To this end, we will combine our data with explanatory landscape analysis, and to build an automated tool to select CMA-ES configurations on the fly. Put differently, we aim at extending the per instance algorithm configuration approach analyzed in [BDSS17] to the modular CMA-ES and towards a non-static selection. Rather than absolute performance gains, our main interests is therefore in identifying the best performing configurations per each function and each phase of the optimization process.…”

Section: Adaptive Configurationsmentioning

confidence: 99%

Online selection of CMA-ES variants

Vermetten¹,

Rijn²,

Bäck³

et al. 2019

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

In the field of evolutionary computation, one of the most challenging topics is algorithm selection. Knowing which heuristics to use for which optimization problem is key to obtaining high-quality solutions. We aim to extend this research topic by taking a first step towards a selection method for adaptive CMA-ES algorithms. We build upon the theoretical work done by van Rijn et al. [PPSN'18], in which the potential of switching between different CMA-ES variants was quantified in the context of a modular CMA-ES framework.We demonstrate in this work that their proposed approach is not very reliable, in that implementing the suggested adaptive configurations does not yield the predicted performance gains. We propose a revised approach, which results in a more robust fit between predicted and actual performance. The adaptive CMA-ES approach obtains performance gains on 18 out of 24 tested functions of the BBOB benchmark, with stable advantages of up to 23%. An analysis of module activation indicates which modules are most crucial for the different phases of optimizing each of the 24 benchmark problems. The module activation also suggests that additional gains are possible when including the (B)IPOP modules, which we have excluded for this present work.The adaptive CMA-ES configurations we use were implemented in the modular CMA-ES framework introduced in [vRWvLB16], which is freely available at [vR18].This framework implements 11 different modules. Of these 11 modules, 9 are binary and 2 are ternary, allowing for a combined total of 4,608 different configurations. The full list of available modules is shown in Table 1.

show abstract

Per instance algorithm configuration of CMA-ES with limited budget

Cited by 82 publications

References 30 publications

Benchmarking discrete optimization heuristics with IOHprofiler

Benchmarking discrete optimization heuristics with IOHprofiler

Automated Algorithm Selection: Survey and Perspectives

Online selection of CMA-ES variants

Contact Info

Product

Resources

About