Prediction of Classifier Training Time Including Parameter Optimization

Abstract

Abstract. Besides the classification performance, the training time is a second important factor that affects the suitability of a classification algorithm regarding an unknown dataset. An algorithm with a slightly lower accuracy is maybe preferred if its training time is significantly lower. Additionally, an estimation of the required training time of a pattern recognition task is very useful if the result has to be available in a certain amount of time. Meta-learning is often used to predict the suitability or performance of classifiers using different learning schemes and features. Especially landmarking features have been used very successfully in the past. The accuracy of simple learners are used to predict the performance of a more sophisticated algorithm. In this work, we investigate the quantitative prediction of the training time for several target classifiers. Different sets of meta-features are evaluated according to their suitability of predicting actual run-times of a parameter optimization by a grid search. Additionally, we adapted the concept of landmarking to time prediction. Instead of their accuracy, the run-time of simple learners are used as feature values. We evaluated the approach on real world datasets from the UCI machine learning repository and StatLib. The run-time of five different classification algorithms are predicted and evaluated using two different performance measures. The promising results show that the approach is able to reasonably predict the training time including a parameter optimization. Furthermore, different sets of meta-features seem to be necessary for different target algorithms in order to achieve the highest prediction performances.

To tune or not to tune: Recommending when to adjust SVM hyper-parameters via meta-learning

“…In these cases, the parameter settings are predicted without actually evaluating the model on the new data set. In [7], [24], MTL is used to estimate the training time of classification algorithms for different hyper-parameter configurations.…”

“…This recommendation system is based on Meta-learning 978-1-4799-1959-8/15/$31.00 @2015 IEEE (MTL) [4] ideas to induce a classification model that, based on the characteristics of a data set, recommends the tuning of the hyper-parameters or the use of default values. MTL has frequently been employed to select [5], rank [6], or predict [7] the performance of ML algorithms on a new data set.…”

Toward a Progress Indicator for Machine Learning Model Building and Data Mining Algorithm Execution

“…Researchers have done much work on predicting machine learning model building time [15, 59–61, 63, 68] and the runtime of a computer program [20, 25, 29, 65, 74], an iterative algorithm on graph data [57], and an algorithm for solving a combinatorial optimization problem [30]. The predicted model building time is often inaccurate and may differ considerably from the actual model building time on a loaded computer.…”

“…The load on such a computer cluster has different characteristics from that in a typical environment (e.g., on a computer in the user’s office), where a machine learning model is built or data mining algorithm is executed. Also, these techniques usually ignore hyper-parameters, whose values can greatly impact machine learning model accuracy and building cost [63, 75], as well as data mining results and algorithm execution cost. Like the case with machine learning, many data mining algorithms have hyper-parameters.…”

“…Another way to offer a trivial progress indicator is to use an existing method to predict a machine learning model’s building time before model building starts [15, 59–61, 63, 68]. If the model building task is predicted to take t seconds and has run for t ′ seconds, the remaining time is projected to be t-t ′ seconds.…”