Benchmarking discrete optimization heuristics with IOHprofiler

Doerr, Benjamin; Ye, Furong; Horesh, Naama; Wang, Hao; Shir, Ofer M.; Bäck, Thomas

doi:10.1016/j.asoc.2019.106027

Cited by 59 publications

(66 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Raw data and extended galleries of tables and fitness trend graphs generated with SOS for some of the studies mentioned in this article are available in [91,92]; • extensive galleries of images obtained with the joint use of SOS, for generating data, and the IOHprofiler in [7], for plotting them, are available in [41].…”

Section: Visual Representation Of Resultsmentioning

confidence: 99%

“…First, the average final fitness value must be computed with values returned by the NA algorithms after the execution of multiple runs. SOS automatically collects the text files containing the information stored in FTrend objects and provides the NP final average values for each algorithm 7 .…”

Section: Assigning Ranksmentioning

confidence: 99%

“…In this light, the research community benefits from platforms featuring a large number of algorithms and benchmark testbed problems, as e.g. the Decision Tree for Optimization software [5], COCO [6], IOHprofiler [7] and jMetal [8], which facilitate this testing. However, the availability of code implementing complete of benchmark suites and popular optimisation algorithms is only one amongst the many "tools" required by an algorithm designer to work efficiently and productively focus 100% on algorithmic issues rather than having to deal with the implementation of ancillary software.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

The SOS Platform: Designing, Tuning and Statistically Benchmarking Optimisation Algorithms

Caraffini¹

2020

Preprint

View full text Add to dashboard Cite

The Stochastic Optimisation Software (SOS) is a Java platform facilitating the algorithmic design process and the evaluation of metaheuristic optimisation algorithms. It reduces the burden of coding miscellaneous methods for dealing with several bothersome and time-demanding tasks such as parameter tuning, implementation of comparison algorithms and testbed problems, collecting and processing data to display results, measuring algorithmic overhead, etc. SOS provides numerous off-the-shelf methods including 1) customised implementations of statistical tests, such as the Wilcoxon Rank-Sum test and the Holm-Bonferroni procedure, for comparing performances of optimisation algorithms and automatically generate result tables in PDF and LaTeX formats; 2) the implementation of an original advanced statistical routine for accurately comparing couples of stochastic optimisation algorithms; 3) the implementation of a novel testbed suite for continuous optimisation, derived from the IEEE CEC 2014 benchmark, allowing for controlled activation of the rotation operator. each testbed function. Moreover, this article comments on the current state of the literature in stochastic optimisation and highlights similarities shared by modern metaheuristics inspired by nature. It is argued that the vast majority of these algorithms are simply a reformulation of the same methods and that metaheuristics for optimisation should be simply treated as stochastic processes with less emphasis on the inspiring metaphor behind them.

show abstract

Section: Visual Representation Of Resultsmentioning

confidence: 99%

Section: Assigning Ranksmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The SOS Platform: Designing, Tuning and Statistically Benchmarking Optimisation Algorithms

Caraffini¹

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Our next steps are as follows. First, we will annotate more data sets, starting with those provided by IOHprofiler [1,3] and those of Nevergrad [2]. Further extensions include a web-based GUI, similar to what is shown in Figure 1.…”

Section: Discussionmentioning

confidence: 99%

“…All annotations were produced with the Apache Jena RDF library in the form of RDF graphs. For each of the 32 algorithms, a 1 The data in the ontology provides the full information on the used problem/dimension/instance/algorithm, as well as the corresponding performance data. In addition, data provenance information has been manually collected and linked to the performance data to trace its origin.…”

Section: Use Casementioning

confidence: 99%

Option

Kostovska

Vermetten²,

Doerr

et al. 2021

Proceedings of the Genetic and Evolutionary Computation Conference Companion

Self Cite

View full text Add to dashboard Cite

Many platforms for benchmarking optimization algorithms offer users the possibility of sharing their experimental data with the purpose of promoting reproducible and reusable research. However, different platforms use different data models and formats, which drastically inhibits identification of relevant data sets, their interpretation, and their interoperability. Consequently, a semantically rich, ontology-based, machine-readable data model is highly desired.We report in this paper on the development of such an ontology, which we name OPTION (OPTImization algorithm benchmarking ONtology). Our ontology provides the vocabulary needed for semantic annotation of the core entities involved in the benchmarking process, such as algorithms, problems, and evaluation measures. It also provides means for automated data integration, improved interoperability, powerful querying capabilities and reasoning, thereby enriching the value of the benchmark data. We demonstrate the utility of OPTION by annotating and querying a corpus of benchmark performance data from the BBOB workshop data -a use case which can be easily extended to cover other benchmarking data collections. CCS CONCEPTS• Computing methodologies → Ontology engineering; Reasoning about belief and knowledge; Continuous space search;

show abstract