Abstract. State-of-the-art algorithms for hard computational problems often expose many parameters that can be modified to improve empirical performance. However, manually exploring the resulting combinatorial space of parameter settings is tedious and tends to lead to unsatisfactory outcomes. Recently, automated approaches for solving this algorithm configuration problem have led to substantial improvements in the state of the art for solving various problems. One promising approach constructs explicit regression models to describe the dependence of target algorithm performance on parameter settings; however, this approach has so far been limited to the optimization of few numerical algorithm parameters on single instances. In this paper, we extend this paradigm for the first time to general algorithm configuration problems, allowing many categorical parameters and optimization for sets of instances. We experimentally validate our new algorithm configuration procedure by optimizing a local search and a tree search solver for the propositional satisfiability problem (SAT), as well as the commercial mixed integer programming (MIP) solver CPLEX. In these experiments, our procedure yielded state-of-the-art performance, and in many cases outperformed the previous best configuration approach.

The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms. We describe an automatic framework for this algorithm configuration problem. More formally, we provide methods for optimizing a target algorithm's performance on a given class of problem instances by varying a set of ordinal and/or categorical parameters. We review a family of local-search-based algorithm configuration procedures and present novel techniques for accelerating them by adaptively limiting the time spent for evaluating individual configurations. We describe the results of a comprehensive experimental evaluation of our methods, based on the configuration of prominent complete and incomplete algorithms for SAT. We also present what is, to our knowledge, the first published work on automatically configuring the CPLEX mixed integer programming solver. All the algorithms we considered had default parameter settings that were manually identified with considerable effort. Nevertheless, using our automated algorithm configuration procedures, we achieved substantial and consistent performance improvements.

It has been widely observed that there is no single "dominant" SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use socalled empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.

Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall. We consider the problem of simultaneously selecting a learning algorithm and setting its hyperparameters. We show that this problem can be addressed by a fully automated approach, leveraging recent innovations in Bayesian optimization. Specifically, we consider feature selection techniques and all machine learning approaches implemented in WEKA's standard distribution, spanning 2 ensemble methods, 10 meta-methods, 28 base learners, and hyperparameter settings for each learner. On each of 21 popular datasets from the UCI repository, the KDD Cup 09, variants of the MNIST dataset and CIFAR-10, we show performance often much better than using standard selection and hyperparameter optimization methods. We hope that our approach will help non-expert users to more effectively identify machine learning algorithms and hyperparameter settings appropriate to their applications, and hence to achieve improved performance.

Perhaps surprisingly, it is possible to predict how long an algorithm will take to run on a previously unseen input, using machine learning techniques to build a model of the algorithm's runtime as a function of problem-specific instance features. Such models have important applications to algorithm analysis, portfolio-based algorithm selection, and the automatic configuration of parameterized algorithms. Over the past decade, a wide variety of techniques have been studied for building such models. Here, we describe extensions and improvements of existing models, new families of models, andperhaps most importantly-a much more thorough treatment of algorithm parameters as model inputs. We also comprehensively describe new and existing features for predicting algorithm runtime for propositional satisfiability (SAT), travelling salesperson (TSP) and mixed integer programming (MIP) problems. We evaluate these innovations through the largest empirical analysis of its kind, comparing to a wide range of runtime modelling techniques from the literature. Our experiments consider 11 algorithms and 35 instance distributions; they also span a very wide range of SAT, MIP, and TSP instances, with the least structured having been generated uniformly at random and the most structured having emerged from real industrial applications. Overall, we demonstrate that our new models yield substantially better runtime predictions than previous approaches in terms of their generalization to new problem instances, to new algorithms from a parameterized space, and to both simultaneously. 102 21 59 109 6E3 1E3 1E3 6E3 Minisat 2.0-HAND 1903 3600 1.25 0.57 0.53 0.52 0.51 74 14 48 79 3E3 96 179 3E3 Minisat 2.0-RAND 2497 3600 0.82 0.39 0.38 0.37 0.37 59 23 52 65 221 35 145 230 Minisat 2.0-INDU 1146 3600 0.94 0.58 0.57 0.55 0.52 222 24 85 231 6E3 1E3 1E3 6E3 Minisat 2.0-SWV-IBM 466 3600 0.85 0.16 0.17 0.17 0.17 98 8.4 59 102 1E3 74 153 1E3 Minisat 2.0-IBM 834 3600 1.1 0.21 0.25 0.21 0.19 130 11 78 136 1E3 74 153 1E3 Minisat 2.0-SWV 0.89 5.32 0.25 0.08 0.09 0.08 0.08 57 4.9 34 59 217 17 123 226 CryptoMinisat-INDU 1921 3600 1.1 0.81 0.73 0.74 0.72 222 24 85 231 6E3 1E3 1E3 6E3 CryptoMinisat-SWV-IBM 873 3600 1.07 0.47 0.5 0.49 0.48 98 8.4 59 102 1081 74 153 1103 CryptoMinisat-IBM 1178 3600 1.2 0.42 0.45 0.42 0.41 130 11 78 136 1081 74 153 1103 CryptoMinisat-SWV 486 3600 0.89 0.51 0.53 0.49 0.51 57 4.9 34 59 217 17 123 226 SPEAR-INDU 1685 3600 1.01 0.67 0.62 0.61 0.58 222 24 85 231 6E3 1E3 1E3 6E3 SPEAR-SWV-IBM 587 3600 0.97 0.38 0.39 0.39 0.38 98 8.4 59 102 1E3 74 153 1E3 SPEAR-IBM 1004 3600 1.18 0.39 0.42 0.42 0.38 130 11 78 136 1E3 74 153 1E3 SPEAR-SWV 60 3600 0.54 0.36 0.34 0.34 0.34 57 4.9 34 59 217 17 123 226 tnm-RANDSAT 568 3600 1.05 0.88 0.97 0.9 0.88 63 26 56 70 221 35 145 230 SAPS-RANDSAT 1019 3600 1 0.67 0.71 0.65 0.66 63 26 56 70 221 35 145 230 CPLEX-BIGMIX 719 3600 0.96 0.84 0.85 0.63 0.64 17 0.13 6.7 23 1E4 6.6 54 1E4 Gurobi-BIGMIX 992 3600 1.31 1.28 1.31 1.19 1.17 17 0.13 6.7 23 1E4 6.6 54 1E4 SCIP-BIGMIX 1153 3600 0.77 0.67 0.72 0.58...