Algorithm Configuration: Learning Policies for the Quick Termination of Poor Performers

Abstract

One way to speed up the algorithm configuration task is to use short runs instead of long runs as much as possible, but without discarding the configurations that eventually do well on the long runs. We consider the problem of selecting the top performing configurations of the Conditional Markov Chain Search (CMCS), a general algorithm schema that includes, for examples, VNS. We investigate how the structure of performance on short tests links with those on long tests, showing that significant differences arise between test domains. We propose a "performance envelope" method to exploit the links; that learns when runs should be terminated, but that automatically adapts to the domain.

Hyperparameter Optimisation with Early Termination of Poor Performers

“…The core idea behind the proposed approach is to instead train each model for only a short time and then extract available information to predict the final score of the model. In other words, following similar work on parameter tuning in optimisation [5], we hypothesise that the final accuracy of a machine learning model can be predicted from its internal state at the early stages of training.…”

“…A recent approach in parameter tuning used within a new optimisation framework, Conditional Markov Chain Search [4], is to predict the performance of a combination of parameters based on information collected after a short run [5]. This can potentially save computational power, however the use of predictions instead of the actually measured performance reduces the quality of the tuning method.…”