Hyper-Parameter Tuning for Support Vector Machines by Estimation of Distribution Algorithms

Padierna, Luis Carlos; Carpío, Martín; Rojas, Alfonso; Soberanes, Héctor José Puga; Baltazar, Rosario; Fraire, Héctor

doi:10.1007/978-3-319-47054-2_53

Cited by 31 publications

(19 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Extensive research have been undertaken in the area of hyper-parameter selection, especially in association with different problems related to machine learning techniques; for an example, see [22][23][24][25], etc. Meta-heuristics is put to work in order to calibrate the hyper-parameters based on zero-th order information.…”

Section: The Problem Of Hyperparameter Calibrationmentioning

confidence: 99%

Efficient Hyper-Parameter Selection in Total Variation-Penalised XCT Reconstruction Using Freund and Shapire’s Hedge Approach

et al. 2020

View full text Add to dashboard Cite

This paper studies the problem of efficiently tuning the hyper-parameters in penalised least-squares reconstruction for XCT. Discovered through the lens of the Compressed Sensing paradigm, penalisation functionals such as Total Variation types of norms, form an essential tool for enforcing structure in inverse problems, a key feature in the case where the number of projections is small as compared to the size of the object to recover. In this paper, we propose a novel hyper-parameter selection approach for total variation (TV)-based reconstruction algorithms, based on a boosting type machine learning procedure initially proposed by Freund and Shapire and called Hedge. The proposed approach is able to select a set of hyper-parameters producing better reconstruction than the traditional Cross-Validation approach, with reduced computational effort. Traditional reconstruction methods based on penalisation can be made more efficient using boosting type methods from machine learning.

show abstract

Section: The Problem Of Hyperparameter Calibrationmentioning

confidence: 99%

Efficient Hyper-Parameter Selection in Total Variation-Penalised XCT Reconstruction Using Freund and Shapire’s Hedge Approach

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Hence, finding suitable SVM HPs is a frequently studied problem [18,34]. SVM HP tuning is commonly modeled as a black-box optimization problem whose objective function is associated with the predictive performance of the SVM induced model.…”

mentioning

confidence: 99%

A meta-learning recommender system for hyperparameter tuning: Predicting when tuning improves SVM classifiers

Mantovani

Rossi

Alcobaça

et al. 2019

Information Sciences

View full text Add to dashboard Cite

For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them. However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default values. This paper proposes a recommender system based on meta-learning to identify exactly when it is better to use default values and when to tune hyperparameters for each new dataset. Besides, an in-depth analysis is performed to understand what they take into account for their decisions, providing useful insights.An extensive analysis of different categories of meta-features, meta-learners, and setups across 156 datasets is performed. Results show that it is possible to accurately predict when tuning will significantly improve the performance of the induced models. The proposed system reduces the time spent on optimization processes, without reducing the predictive performance of the induced models (when compared with the ones obtained using tuned hyperparameters). We also explain the decision-making process of the meta-learners in terms of linear separability-based hypotheses. Although this analysis is focused on the tuning of Support Vector Machines, it can also be applied to other algorithms, as shown in experiments performed with decision trees.

show abstract

“…However, despite their good predictive performance, they are highly sensitive to their HP values. Due to that, SVM HP tuning is still widely studied in literature (BRAGA et al, 2013;DUARTE;WAINER, 2017;HORN et al, 2016;PADIERNA et al, 2017). As mentioned before, different tuning techniques may obtain different HP settings from different hyperspace regions.…”

Section: Tuning Of Svmsmentioning

confidence: 99%

“…In this chapter, the HP profile of SVMs was investigated. SVMs are highly sensitive to their HP values, and thus still widely studied in recent literature (PADIERNA et al, 2017;LORENA et al, 2018), but few of them compare different techniques when performing HP tuning (HORN et al, 2016).…”

Section: Chapter Remarksmentioning

confidence: 99%

See 1 more Smart Citation

Use of meta-learning for hypeparameter tuning of classification problems

Mantovani¹

View full text Add to dashboard Cite

Machine learning solutions have been successfully used to solve many simple and complex problems. However, their development process still relies on human experts to perform tasks such as data preprocessing, feature engineering and model selection. As the complexity of these tasks increases, so does the demand for automated solutions, namely Automated Machine Learning (AutoML). Most algorithms employed in these systems have hyperparameters whose configuration may directly affect their predictive performance. Therefore, hyperparameter tuning is a recurring task in AutoML systems. This thesis investigated how to efficiently automate hyperparameter tuning by means of Meta-learning. To this end, large-scale experiments were performed tuning the hyperparameters of different classification algorithms, and an enhanced experimental methodology was adopted throughout the thesis to explore and learn the hyperparameter profiles for different classification algorithms. The results also showed that in many cases the default hyperparameter settings induced models that are on par with those obtained by tuning. Hence, a new Meta-learning recommender system was proposed to identify when it is better to use default values and when to tune classification algorithms for each new dataset. The proposed system is capable of generalizing several learning processes into a single modular framework, along with the possibility of assigning different algorithms. Furthermore, a descriptive analysis of model predictions is used to identify which data characteristics affect the necessity for tuning in each one of the algorithms investigated in the thesis. Experimental results also demonstrated that the proposed recommender system reduced the time spent on optimization processes, without reducing the predictive performance of the induced models. Depending on the target algorithm, the Meta-learning recommender system can statistically outperform the baselines. The significance of these results opens a number of new avenues for future work.

show abstract

Hyper-Parameter Tuning for Support Vector Machines by Estimation of Distribution Algorithms

Cited by 31 publications

References 9 publications

Efficient Hyper-Parameter Selection in Total Variation-Penalised XCT Reconstruction Using Freund and Shapire’s Hedge Approach

Efficient Hyper-Parameter Selection in Total Variation-Penalised XCT Reconstruction Using Freund and Shapire’s Hedge Approach

A meta-learning recommender system for hyperparameter tuning: Predicting when tuning improves SVM classifiers

Use of meta-learning for hypeparameter tuning of classification problems

Contact Info

Product

Resources

About