Approximating parameterized convex optimization problems

Giesen, Joachim; Jäggi, Martin; Laue, Sören

doi:10.1145/2390176.2390186

Cited by 27 publications

(35 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ν + and ν − . Thus, an approximate two-dimensional solution surface algorithm should be designed for tackling this parametric nonlinear optimization problem [30]. Similarly, an approximate multidimensional solution surface algorithm should be designed for CSHL-SVM [11] which is nonlinear w.r.t.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Cross Validation Through Two-Dimensional Solution Surface for Cost-Sensitive SVM

Sheng

Tay

et al. 2017

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Model selection plays an important role in cost-sensitive SVM (CS-SVM). It has been proven that the global minimum cross validation (CV) error can be efficiently computed based on the solution path for one parameter learning problems. However, it is a challenge to obtain the global minimum CV error for CS-SVM based on one-dimensional solution path and traditional grid search, because CS-SVM is with two regularization parameters. In this paper, we propose a solution and error surfaces based CV approach (CV-SES). More specifically, we first compute a two-dimensional solution surface for CS-SVM based on a bi-parameter space partition algorithm, which can fit solutions of CS-SVM for all values of both regularization parameters. Then, we compute a two-dimensional validation error surface for each CV fold, which can fit validation errors of CS-SVM for all values of both regularization parameters. Finally, we obtain the CV error surface by superposing K validation error surfaces, which can find the global minimum CV error of CS-SVM. Experiments are conducted on seven datasets for cost sensitive learning and on four datasets for imbalanced learning. Experimental results not only show that our proposed CV-SES has a better generalization ability than CS-SVM with various hybrids between grid search and solution path methods, and than recent proposed cost-sensitive hinge loss SVM with three-dimensional grid search, but also show that CV-SES uses less running time.

show abstract

Section: Resultsmentioning

confidence: 99%

“…ν + and ν − . Thus, an approximate solution surface algorithm should be designed for tackling this parametric nonlinear optimization problem [30], which is beyond the scope of this paper.…”

Section: : Partition the Parameter Spacementioning

confidence: 99%

Cross Validation Through Two-Dimensional Solution Surface for Cost-Sensitive SVM

Sheng

Tay

et al. 2017

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…and β is the smallest real number greater than β verifying this property. Notice also that (35) shows that β is the smallest real number greater than β such that, for some i,…”

Section: Remark 32mentioning

confidence: 99%

“…We do this by formulating a bilevel optimization problem, where the feasible set is the (properly) Pareto set of the vectorial elastic net problem. The numerical approximation of the so-called regularization path has been done, e.g., by Giesen et al [35,36]. Based on the fact that the regularization path for a fixed L 2 -regularization path is piecewise linear, in this paper we propose a new exact algorithm, which we call Ensalg (Elastic Net Solution Algorithm), to compute it efficiently.…”

Section: Introductionmentioning

confidence: 99%

Post-Pareto Analysis and a New Algorithm for the Optimal Parameter Tuning of the Elastic Net

Bonnel

Schneider

2019

J Optim Theory Appl

View full text Add to dashboard Cite

The paper deals with the optimal parameter tuning for the elastic net problem. This process is formulated as an optimization problem over a Pareto set. The Pareto set is associated with a convex multi-objective optimization problem, and, based on the scalarization theorem, we give a parametrical representation of it. Thus, the problem becomes a bilevel optimization with a unique response of the follower (strong Stackelberg game). Then, we apply this strategy to the parameter tuning for the elastic net problem. We propose a new algorithm called Ensalg to compute the optimal regularization path of the elastic net w.r.t. the sparsity-inducing term in the objective. In contrast to existing algorithms, our method can also deal with the so-called "many-at-a-time" case, where more than one variable becomes zero at the same time and/or changes from zero. In examples involving real-world data, we demonstrate the effectiveness of the algorithm.

show abstract

“…These shortcomings that also show up in practice sparked interest in the development of more robust and efficient approximate path algorithms (Friedman et al 2007;Rosset 2004). By now numerically robust, approximate regularization path following algorithms are known for many problems including support vector machines (Giesen, Jaggi, and Laue 2012a;Giesen et al 2012), the Lasso (Mairal and Yu 2012), and regularized matrix factorization and completion problems (Giesen, Jaggi, and Laue 2012b;Giesen et al 2012). For a prescribed accuracy ε > 0, these algorithms compute a piecewise constant approximation of the solution path, which is called an ε-approximate solution gamut.…”

Section: Introductionmentioning

confidence: 99%

Using Benson’s Algorithm for Regularization Parameter Tracking

Giesen

Laue

Löhne³

et al. 2019

AAAI

Self Cite

View full text Add to dashboard Cite

Regularized loss minimization, where a statistical model is obtained from minimizing the sum of a loss function and weighted regularization terms, is still in widespread use in machine learning. The statistical performance of the resulting models depends on the choice of weights (regularization parameters) that are typically tuned by cross-validation. For finding the best regularization parameters, the regularized minimization problem needs to be solved for the whole parameter domain. A practically more feasible approach is covering the parameter domain with approximate solutions of the loss minimization problem for some prescribed approximation accuracy. The problem of computing such a covering is known as the approximate solution gamut problem. Existing algorithms for the solution gamut problem suffer from several problems. For instance, they require a grid on the parameter domain whose spacing is difficult to determine in practice, and they are not generic in the sense that they rely on problem specific plug-in functions. Here, we show that a well-known algorithm from vector optimization, namely the Benson algorithm, can be used directly for computing approximate solution gamuts while avoiding the problems of existing algorithms. Experiments for the Elastic Net on real world data sets demonstrate the effectiveness of Benson’s algorithm for regularization parameter tracking.

show abstract

Approximating parameterized convex optimization problems

Cited by 27 publications

References 14 publications

Cross Validation Through Two-Dimensional Solution Surface for Cost-Sensitive SVM

Cross Validation Through Two-Dimensional Solution Surface for Cost-Sensitive SVM

Post-Pareto Analysis and a New Algorithm for the Optimal Parameter Tuning of the Elastic Net

Using Benson’s Algorithm for Regularization Parameter Tracking

Contact Info

Product

Resources

About