Experimental Analysis of Optimization Algorithms: Tuning and Beyond

Abstract

This chapter comprises the essence of several years of tutorials the authors gave on experimental research in evolutionary computation. We highlight the renaissance of experimental techniques also in other fields to especially focus on the specific conditions of experimental research in computer science, or more concrete, metaheuristic optimization. The experimental setup is discussed together with the pitfalls awaiting the unexperienced (and sometimes even the experienced). We present a severity criterion as a meta-statistical concept for evaluating statistical inferences, which can be used to avoid fallacies, i.e., misconceptions resulting from incorrect reasoning in argumentation caused by floor or ceiling effects. The sequential parameter optimization is discussed as a meta-statistical framework which integrates concepts such as severity. Parameter tuning is considered as a relatively new tool in method design and analysis, and it leads to the question of adaptability of optimization algorithms. Another branch of experimentation aims for attaining more concrete problem knowledge, we may term it 'exploratory landscape analysis', containing sample and visualization techniques that are often applied but not seen as being a methodological contribution. However, this chapter is not only a renarration of well known facts. We also try a look into the future to estimate what the hot topics of methodological research will be in the next years and what changes we may expect for the whole community.

A scatter search algorithm for the distributed permutation flowshop scheduling problem

“…Actually, a full factorial design analyzed by means of ANOVA can be considered as the first step in an algorithm calibration. For more exhaustive approaches, the reader is referred to Bartz-Beielstein et al (2010) where advanced techniques are shown. The reason behind our choice of a simple calibration is none other than to avoid an unfair comparison with existing approaches.…”

Best practices for comparing optimization algorithms

“…Examples of these efforts include the machine learning based method proposed in [BHBG07], CPLEX automatic tuning tool [CPL14], use of derivative-free optimization [AO06], ParamILS [HHLBS09], and the procedure proposed in [HHLB10] for mixed integer programming solvers. Similarly, some of the tuning techniques for non-deterministic methods include sequential parameter optimization (SPO) [BBLP05,BBP14], relevance and calibration approach [NE06], and F-Race [Bir09].…”

From evolutionary computation to the evolution of things

“…These can customize an initial algorithm setup for a given problem off-line (before the run), or on-line (during the run) 63 . Techniques such as automated parameter tuning 64,65,66,67 and adaptive parameter control continue to make advances in this area 68,69,70,71 .…”