Tuning L1-SVM Hyperparameters with Modified Radius Margin Bounds and Simulated Annealing

Abstract

Abstract. In the design of support vector machines an important step is to select the optimal hyperparameters. One of the most used estimators of the performance is the Radius-Margin bound. Some modifications of this bound have been made to adapt it to soft margin problems, giving a convex optimization problem for the L2 soft margin formulation. However, it is still interesting to consider the L1 case due to the reduction in the support vector number. There have been some proposals to adapt the Radius-Margin bound to the L1 case, but the use of gradient descent to test them is not possible in some of them because these bounds are not differentiable. In this work we propose to use simulated annealing as a method to find the optimal hyperparameters when the bounds are not differentiable, have multiple local minima or the kernel is not differentiable with respect to its hyperparameters.

Spatial prediction of soil properties in temperate mountain regions using support vector regression

“…The Bessel kernel requires the optimization of 3 hyper-parameters, thus grid search should be performed in a 3-dimensional space, which makes the grid search cumbersome. To avoid performing a full grid search, a Simulated Annealing (SA) approach (Kirkpatrick et al, 1983;Acevedo et al, 2007), can be applied to parameters optimization. The SA approach although time consuming is faster than a complete grid search, thus reducing the time required to fit the model.…”

Continuous hyperparameter optimization for large-scale recommender systems

“…While some techniques are algorithm-specific such as [14] and [15], some global optimization approaches can be applied to any algorithms, including collaborative filtering algorithms. Two widely used global optimization approaches are grid search and random search [8].…”