Bilevel model selection for support vector
                    machines

Kunapuli, Gautam; Bennett, Kristin P.; Hu, Jing; Pang, Jong-Shi

doi:10.1090/crmp/045/07

Cited by 20 publications

(18 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They are nonconvex and nonsmooth, but their number remains in the size of w and no new variables are introduced. This hopefully permits better scalability than MPEC bilevel attempts (Bennett et al 2006;Kunapuli et al 2008). A penalty approach is used to bring these constraints into the outer-level objective.…”

Section: Explicit Model Selection Methodsmentioning

confidence: 99%

“…Model selection via T -fold cross-validation can be written compactly as single bilevel program (Bennett et al 2006;Kunapuli et al 2008). The inner-level problems minimize the regularized training error to determine the best function for the given hyperparameters for each fold.…”

Section: Bilevel Cross-validation Problemmentioning

confidence: 99%

“…If the smooth SVM is used, the KKT conditions include complementarity constraints. These are also known as equilibrium constraints, so the problem becomes a Mathematical Programming with Equilibrium Constraints (MPEC) and this formed the basis of the prior MPEC approach by Bennett et al (2006) and Kunapuli et al (2008). The new nonsmooth bilevel approach replaces the inner-level problem with a nonlinear constraint.…”

Section: Bilevel Cross-validation Problemmentioning

confidence: 99%

“…The first explicit bilevel programming approach used the smoothed version of the SVM problem and treated the problem as a mathematical program with equilibrium constraints (MPEC) (Bennett et al 2006;Kunapuli et al 2008). The MPEC approach can handle many hyperparameters and generates good generalization errors when compared with grid search.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Model selection for primal SVM

2011

View full text Add to dashboard Cite

This paper introduces two types of nonsmooth optimization methods for selecting model hyperparameters in primal SVM models based on cross-validation. Unlike common grid search approaches for model selection, these approaches are scalable both in the number of hyperparameters and number of data points. Taking inspiration from linear-time primal SVM algorithms, scalability in model selection is achieved by directly working with the primal variables without introducing any dual variables. The proposed implicit primal gradient descent (ImpGrad) method can utilize existing SVM solvers. Unlike prior methods for gradient descent in hyperparameters space, all work is done in the primal space so no inversion of the kernel matrix is required. The proposed explicit penalized bilevel programming (PBP) approach optimizes both the hyperparameters and parameters simultaneously. It solves the original cross-validation problem by solving a series of least squares regression problems with simple constraints in both the hyperparameter and parameter space. Computational results on least squares support vector regression problems with multiple hyperparameters establish that both the implicit and explicit methods perform quite well in terms of generalization and computational time. These methods are directly applicable to other learning tasks with differentiable loss functions and regularization functions. Both the implicit and explicit algorithms investigated represent powerful new approaches to solving large bilevel programs involving nonsmooth loss functions.

show abstract

Section: Explicit Model Selection Methodsmentioning

confidence: 99%

Section: Bilevel Cross-validation Problemmentioning

confidence: 99%

Section: Bilevel Cross-validation Problemmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Model selection for primal SVM

2011

View full text Add to dashboard Cite

show abstract

“…A similar non-convex optimization problem has been tackled by Kunapuli et al (2007). The fundamental difference between their setting and ours is that they had a single bi-level optimization problem, while we have a family of such problems, parameterized by τ .…”

Section: E(l τ (Y − C)|x) = Quantile τ Of P (Y |X) (2)mentioning

confidence: 99%

Bi-level path following for cross validated solution of kernel quantile regression

Rosset

2008

Proceedings of the 25th International Conference on Machine Learning - ICML '08

View full text Add to dashboard Cite

Modeling of conditional quantiles requires specification of the quantile being estimated and can thus be viewed as a parameterized predictive modeling problem. Quantile loss is typically used, and it is indeed parameterized by a quantile parameter. In this paper we show how to follow the path of cross validated solutions to regularized kernel quantile regression. Even though the bi-level optimization problem we encounter for every quantile is non-convex, the manner in which the optimal cross-validated solution evolves with the parameter of the loss function allows tracking of this solution. We prove this property, construct the resulting algorithm, and demonstrate it on data. This algorithm allows us to efficiently solve the whole family of bi-level problems.

show abstract

Algorithms for Simple Bilevel Programming

Dutta

Pandit

2020

Springer Optimization and Its Applications

View full text Add to dashboard Cite

We develop new algorithms for Riemannian bilevel optimization. We focus in particular on batch and stochastic gradient-based methods, with the explicit goal of avoiding second-order information such as Riemannian hyper-gradients. We propose and analyze RF 2 SA, a method that leverages first-order gradient information to navigate the complex geometry of Riemannian manifolds efficiently. Notably, RF 2 SA is a singleloop algorithm, and thus easier to implement and use. Under various setups, including stochastic optimization, we provide explicit convergence rates for reaching ϵ-stationary points. We also address the challenge of optimizing over Riemannian manifolds with constraints by adjusting the multiplier in the Lagrangian, ensuring convergence to the desired solution without requiring access to second-order derivatives.

show abstract

Bilevel model selection for support vector machines

Cited by 20 publications

References 46 publications

Model selection for primal SVM

Model selection for primal SVM

Bi-level path following for cross validated solution of kernel quantile regression

Algorithms for Simple Bilevel Programming

Contact Info

Product

Resources

About