An Experimental Study of Adaptive Capping in irace

Abstract

The irace package is a widely used for automatic algorithm configuration and implements various iterated racing procedures. The original irace was designed for the optimisation of the solution quality reached within a given running time, a situation frequently arising when configuring algorithms such as stochastic local search procedures. However, when applied to configuration scenarios that involve minimising the running time of a given target algorithm, irace falls short of reaching the performance of other general-purpose configuration approaches, since it tends to spend too much time evaluating poor configurations. In this article, we improve the efficacy of irace in running time minimisation by integrating an adaptive capping mechanism into irace, inspired by the one used by ParamILS. We demonstrate that the resulting iracecap reaches performance levels competitive with those of state-of-the-art algorithm configurators that have been designed to perform well on running time minimisation scenarios. We also investigate the behaviour of iracecap in detail and contrast different ways of integrating adaptive capping.

Automated Algorithm Selection: Survey and Perspectives

“…Realistic algorithm configuration scenarios typically involve tens to hundreds of parameters (see, e.g., Hutter et al, 2009Hutter et al, , 2011López-Ibáñez et al, 2016;Ansótegui et al, 2015;Thornton et al, 2013;Kotthoff et al, 2017). Therefore, per-set algorithm selection techniques are typically not directly applicable to algorithm configuration, although racing techniques can be extended to work well in this case (see, e.g., López-Ibáñez et al, 2016;Pérez Cáceres et al, 2017). Per-set algorithm configuration is closely related to hyperparameter optimisation in machine learning; the main difference is that in algorithm configuration, performance is to be optimised on a possibly diverse set of problem instances, which often requires trading off performance on some instance against that achieved on others.…”

A novel hybrid PSO-based metaheuristic for costly portfolio selection problems

“…At each iteration irace selects the configurations to be discarded, by applying statistical tests such as the Friedman's non-parametric two-way analysis of variance by ranks, its extensions or the paired sample t-test. In addition, irace adopts an adaptive capping mechanism, which reduces the time wasted in the evaluation of poorly performing configurations, by bounding the maximum running time permitted for each such evaluation (Cáceres et al 2017).…”

Towards Portfolios of Streamlined Constraint Models: A Case Study with the Balanced Academic Curriculum Problem

“…Poorly-performing streamliner/model combinations can timeout on several instances and consume a large amount of the training budget. Hence, we employ various racing techniques [3,7,9,25] to terminate poorly-performing combinations early. The streamliner search can then allocate more time to evaluate promising streamliners and gain a more accurate estimation of their performance.…”

“…Adaptive capping is another technique in automated algorithm configuration to terminate poorly-performing algorithm configurations early when optimising running time of a parameterised algorithm. It was first proposed in the local search-based automated algorithm configuration tool ParamILS [25], and was later integrated into irace [9]. The technique has been shown to significantly speed up the search for the best algorithm configurations in many cases [9,24,25].…”

“…It was first proposed in the local search-based automated algorithm configuration tool ParamILS [25], and was later integrated into irace [9]. The technique has been shown to significantly speed up the search for the best algorithm configurations in many cases [9,24,25]. Our streamliner search makes use of the adaptive capping mechanism of irace.…”