Sequential Model-Based Parameter Optimization: an Experimental Investigation of Automated and Interactive Approaches

Abstract

This work experimentally investigates model-based approaches for optimizing the performance of parameterized randomized algorithms. Such approaches build a response surface model and use this model for finding good parameter settings of the given algorithm. We evaluated two methods from the literature that are based on Gaussian process models: sequential parameter optimization (SPO) (Bartz-Beielstein et al, 2005) and sequential Kriging optimization (SKO) (Huang et al, 2006). SPO performed better "out-of-the-box," whereas SKO was competitive when response values were log transformed. We then investigated key design decisions within the SPO paradigm, characterizing the performance consequences of each. Based on these findings, we propose a new version of SPO, dubbed SPO + , which extends SPO with a novel intensification procedure and a log-transformed objective function. In a domain for which performance results for other (model-free) parameter optimization approaches are available, we demonstrate that SPO + achieves state-of-the-art performance. Finally, we compare this automated parameter tuning approach to an interactive, manual process that makes use of classical regression techniques. This interactive approach is particularly useful when only a relatively small number of parameter configurations can be evaluated. Because it can relatively quickly draw attention to important parameters and parameter interactions, it can help experts gain insights into the parameter response of a given algorithm and identify reasonable parameter settings.

Optimization of biogas production with computational intelligence a comparative study

“…SPOT was successfully applied to numerous optimization algorithms, especially in the field of evolutionary computation, e.g., evolution strategies, particle swarm optimization or genetic programming. It was applied in various domains, e.g., machine engineering, the aerospace industry, bioinformatics, CI and games as well as in fundamental research (Beume et al, 2008;Henrich et al, 2008;Lucas and Roosen, 2009;Preuss et al, 2007;Fialho et al, 2009;Fober et al, 2009;Stoean et al, 2009;Hutter et al, 2010).…”

Experimental Analysis of Optimization Algorithms: Tuning and Beyond

“…SPOT was successfully applied in the fields of bioinformatics [79,33,32], environmental engineering [48,30], shipbuilding [72], fuzzy logic [82], multimodal optimization [68], statistical analysis of algorithms [50,78], multicriteria optimization [80], genetic programming [51], particle swarm optimization [9,49], automated and manual parameter tuning [31,74,42,43], graph drawing [77,65], aerospace and shipbuilding industry [63], mechanical engineering [56], and chemical engineering [39]. Bartz-Beielstein [3] collects publications related to the sequential parameter optimization.…”

“…This is not the case for more sophisticated prediction models, such as neural networks or Gaussian process models. Furthermore, as demonstrated in [42], it is possible to obtain competitive results using such simple models. Nevertheless, in principle, more complex regression models could be used in the context of the interactive sequential parameter optimization approach.…”

A heuristic approach to schedule reoptimization in the context of interactive optimization

“…The first type of method uses interaction in order to improve the performance of the optimization process. This form of interaction appears, for instance, in human-guided search approaches [13], as well as interactive parameter tuning [12]. In the second type of interactive method, the objective is to "enrich" the optimization model.…”