Dynamic Algorithm Selection Using Reinforcement Learning

Abstract

It is often the case that many algorithms exist to solve a single problem, each possessing different performance characteristics. The usual approach in this situation is to manually select the algorithm which has the best average performance. However, this strategy has drawbacks in cases where the optimal algorithm changes during an invocation of the program, in response to changes in the program's state and the computational environment. This paper presents a prototype tool that uses reinforcement learning to guide algorithm selection at runtime, matching the algorithm used to the current state of the computation. The tool is applied to a simulation similar to those used in some computational chemistry problems. It is shown that the low dimensionality of the problem enables the optimal choice of algorithm to be determined quickly, and that the learning system can react rapidly to phase changes in the target program.

A generic adaptive simulation algorithm for component-based simulation systems

“…Along these lines, Armstrong et al [1] introduce a general scheme for adaptive algorithms that employs reinforcement learning in the same manner: an 'optimization system' collects sensor data (i.e., features of the input data), invokes a reinforcement learning algorithm, and executes its decisions (i.e., which algorithm to use). The (negated) execution time of the selected algorithm serves as reward.…”

“…To do so, we follow [1] and introduce a preprocessor function p : Σ * → S that aggregates a trajectory τ ∈ Σ * to a state s ∈ S. As not all states of the environment are distinguishable anymore, we now deal with a partially observable Markov decision process.…”

“…Firstly, we integrate it into a component-based simulation system that offers many more adaptation options. Even rather simple learning schemes may improve the performance when there are just two (as in [1]) or three (as in [23]) possible actions, but it is still unclear how well this scales to more actions. Secondly, our approach is tailored toward simulations.…”

Algorithm Portfolios for Noisy Optimization: Compare Solvers Early

“…[14,15] propose 50% for the best solver, 25% for the second best, and so on. Some selection solvers [15,2] do not need a separate training phase, and performs entirely online solver selection; a weakness of this approach is that it is only possible when a large enough budget is available, so that the training phase has a minor cost. At the moment, the case of portfolios of noisy optimization solvers has not been discussed in the literature.…”

Online model racing based on extreme performance

“…This is a typical online model selection problem, for which, many algorithms have been developed and applied in a wide variety of algorithm-problem settings. In [13,1], for instance, model selection is formulated as a Markov Decision Process, and reinforcement learning is used to find the optimal algorithm scheduling strategy. In [9,6], multi-armed bandit algorithms are employed to find the optimal assignment of available computational resources.…”