Identifying Key Algorithm Parameters and Instance Features Using Forward Selection

Abstract

Abstract. Most state-of-the-art algorithms for large scale optimization expose free parameters, giving rise to combinatorial spaces of possible configurations. Typically, these spaces are hard for humans to understand. In this work, we study a model-based approach for identifying a small set of both algorithm parameters and instance features that suffices for predicting empirical algorithm performance well. Our empirical analyses on a wide variety of hard combinatorial problem benchmarks (spanning SAT, MIP, and TSP) show that-for parameter configurations sampled uniformly at random-very good performance predictions can typically be obtained based on just two key parameters, and that similarly, few instance features and algorithm parameters suffice to predict the most salient algorithm performance characteristics in the combined configuration/feature space. We also use these models to identify settings of these key parameters that are predicted to achieve the best overall performance, both on average across instances and in an instance-specific way. This serves as a further way of evaluating model quality and also provides a tool for further understanding the parameter space. We provide software for carrying out this analysis on arbitrary problem domains and hope that it will help algorithm developers gain insights into the key parameters of their algorithms, the key features of their instances, and their interactions.

Analysing differences between algorithm configurations through ablation

“…This allows algorithm developers or users to find a minimal set of parameter modifications from a given default configuration, while maintaining most or all of the performance gains achieved by automated algorithm configuration. We believe that this approach can be complementary to the recent model-based techniques of Hutter et al [12,13], as the local information provided by our approach can strengthen and validate (or invalidate) the results obtained with those techniques. We also believe that there are ways to combine the two lines of work (see Section 6).…”

“…Very recently, Hutter et al have been using model-based techniques to investigate the problems of parameter importance and parameter interaction directly, using forward selection [13] and functional ANOVA [12]. Both approaches require an initial data-gathering step to obtain algorithm performance data, which is then partitioned into training and test sets.…”

“…Both approaches require an initial data-gathering step to obtain algorithm performance data, which is then partitioned into training and test sets. In [13], this data was obtained by sampling 1 000 -10 000 pairs of configurations and instances uniformly at random, while in [12] 10 000 randomly sampled runs were combined with the algorithm runs performed during 25 executions of the SMAC configurator (62 861 -1 354 189 additional runs 1 ).…”

“…In the forward selection approach [13] this performance data is used to iteratively build a regression model by greedily adding, at each iteration, the parameter or instance feature which results in a model with the lowest root mean squared error on the validation set. The work in [12], on the other hand, introduces an efficient technique for applying functional ANOVA to random forest models.…”

“…We chose to analyze the industrial SAT solver SPEAR 1.2.1, winner of one category of the 2007 Satisfiability Modulo Theories Competition [9]; SPEAR has 26 configurable parameters, creating a space of 8.34 × 10 17 configurations. SPEAR has also been used in two recent investigations of parameter importance using forward selection [13] and functional ANOVA [12]. We analyzed the performance of SPEAR on the SWV software verification instance set used in several previous investigations.…”