We develop a hyperparameter optimisation algorithm, Automated Budget Constrained Training, which balances the quality of a model with the computational cost required to tune it. The relationship between hyperparameters, model quality and computational cost must be learnt and this learning is incorporated directly into the optimisation problem. At each training epoch, the algorithm decides whether to terminate or continue training, and, in the latter case, what values of hyperparameters to use. This decision weighs optimally potential improvements in the quality with the additional training time and the uncertainty about the learnt quantities. The performance of our algorithm is verified on a number of machine learning problems encompassing random forests and neural networks. Our approach is rooted in the theory of Markov decision processes with partial information and we develop a numerical method to compute the value function and an optimal strategy.

We develop a hyperparameter optimisation algorithm, Automated Budget Constrained Training (AutoBCT), which balances the quality of a model with the computational cost required to tune it. The relationship between hyperparameters, model quality and computational cost must be learnt and this learning is incorporated directly into the optimisation problem. At each training epoch, the algorithm decides whether to terminate or continue training, and, in the latter case, what values of hyperparameters to use. This decision weighs optimally potential improvements in the quality with the additional training time and the uncertainty about the learnt quantities. The performance of our algorithm is verified on a number of machine learning problems encompassing random forests and neural networks. Our approach is rooted in the theory of Markov decision processes with partial information and we develop a numerical method to compute the value function and an optimal strategy.

We present Wavelet Monte Carlo (WMC), a new method for generating independent samples from complex target distributions. The methodology is based on wavelet decomposition of the difference between the target density and a user-specified initial density, and exploits both wavelet theory and survival analysis. In practice, WMC can process only a finite range of wavelet scales. We prove that the resulting $$L_1$$ L 1 approximation error converges to zero geometrically as the scale range tends to $$(-\infty ,+\infty )$$ ( - ∞ , + ∞ ) . This provides a principled approach to trading off accuracy against computational efficiency. We offer practical suggestions for addressing some issues of implementation, but further development is needed for a computationally efficient methodology. We illustrate the methodology in one- and two-dimensional examples, and discuss challenges and opportunities for application in higher dimensions.