Bi-level path following for cross validated solution of kernel quantile regression

Rosset, Saharon

doi:10.1145/1390156.1390262

Cited by 8 publications

(21 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cross-validation could be replaced by the computation of a suitable model selection criterion, as done in previous work (Li and Zhu 2008). For quantile regression, Rosset (2008) proposes a bi-level solution path algorithm for varying regularization parameter and varying quantile τ , thus elegantly avoiding the problem of quantile crossing (Koenker 2005). An extension of the path algorithm of Sect.…”

Section: Discussionmentioning

confidence: 99%

The structured elastic net for quantile regression and support vector classification

Slawski

2010

Stat Comput

View full text Add to dashboard Cite

In view of its ongoing importance for a variety of practical applications, feature selection via 1 -regularization methods like the lasso has been subject to extensive theoretical as well empirical investigations. Despite its popularity, mere 1 -regularization has been criticized for being inadequate or ineffective, notably in situations in which additional structural knowledge about the predictors should be taken into account. This has stimulated the development of either systematically different regularization methods or double regularization approaches which combine 1 -regularization with a second kind of regularization designed to capture additional problem-specific structure. One instance thereof is the 'structured elastic net', a generalization of the proposal in Zou and Hastie (J. R. Stat. Soc. Ser. B 67:301-320, 2005), studied in Slawski et al. (Ann. Appl. Stat. 4(2):1056-1080, 2010) for the class of generalized linear models.In this paper, we elaborate on the structured elastic net regularizer in conjunction with two important loss functions, the check loss of quantile regression and the hinge loss of support vector classification. Solution paths algorithms are developed which compute the whole range of solutions as one regularization parameter varies and the second one is kept fixed.The methodology and practical performance of our approach is illustrated by means of case studies from image classification and climate science.

show abstract

Section: Discussionmentioning

confidence: 99%

The structured elastic net for quantile regression and support vector classification

Slawski

2010

Stat Comput

View full text Add to dashboard Cite

show abstract

“…In this section, we use the CCPRs of both (λ, η)-SVM and 2C-SVM to explore the region of [0, 1] × [0, 1] for the (λ, η) parameter space and the lower triangle region of [0, 1] × [0, 1] for the (C + , C − ) parameter space respectively. Once the 1.5 square units have been completely covered by the CCPRs, we can fit solutions of CS-SVM for all values of (C + , C − ) according to (14)- (16) and (20)- (21). This means that the complete two-dimensional solution surface of CS-SVM is determined.…”

Section: Bi-parameter Space Partition Algorithmmentioning

confidence: 99%

“…To overcome the first difficulty, solution path algorithms were proposed for many learning models [12], [13], [14], [15], [16], [17], [18], [20], to fit the entire solutions for every value of the parameter, which avoids training the classifier many times under different parameter settings. Specifically, Hastie et al [12] proposed an solution path approach for C-SVM.…”

Section: Introductionmentioning

confidence: 99%

“…It should be noted that there are several articles involving solution paths for two-parameter problems ε-SVR [15], 2C-SVM [14], and quantile regression [14]. Although they are computed in a twoparameter space, all are essentially one-parametric solution paths and none of them can fit solutions for all values of a parameter pair.…”

Section: Introductionmentioning

confidence: 99%

“…This method is implemented (denoted as SP+GS in our paper) to compare with our algorithm. 3) Rosset [14] handled the problem (generating a set of models w.r.t. τ , by selecting the best regularization parameter λ(τ ) for every value of τ ) using a bi-level program with optimizing one parameter λ.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Cross Validation Through Two-Dimensional Solution Surface for Cost-Sensitive SVM

Sheng

Tay

et al. 2017

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Model selection plays an important role in cost-sensitive SVM (CS-SVM). It has been proven that the global minimum cross validation (CV) error can be efficiently computed based on the solution path for one parameter learning problems. However, it is a challenge to obtain the global minimum CV error for CS-SVM based on one-dimensional solution path and traditional grid search, because CS-SVM is with two regularization parameters. In this paper, we propose a solution and error surfaces based CV approach (CV-SES). More specifically, we first compute a two-dimensional solution surface for CS-SVM based on a bi-parameter space partition algorithm, which can fit solutions of CS-SVM for all values of both regularization parameters. Then, we compute a two-dimensional validation error surface for each CV fold, which can fit validation errors of CS-SVM for all values of both regularization parameters. Finally, we obtain the CV error surface by superposing K validation error surfaces, which can find the global minimum CV error of CS-SVM. Experiments are conducted on seven datasets for cost sensitive learning and on four datasets for imbalanced learning. Experimental results not only show that our proposed CV-SES has a better generalization ability than CS-SVM with various hybrids between grid search and solution path methods, and than recent proposed cost-sensitive hinge loss SVM with three-dimensional grid search, but also show that CV-SES uses less running time.

show abstract

Multi-parametric solution-path algorithm for instance-weighted support vector machines

Karasuyama¹,

Harada²,

Sugiyama³

et al. 2012

Mach Learn

View full text Add to dashboard Cite

An instance-weighted variant of the support vector machine (SVM) has attracted considerable attention recently since they are useful in various machine learning tasks such as non-stationary data analysis, heteroscedastic data modeling, transfer learning, learning to rank, and transduction. An important challenge in these scenarios is to overcome the computational bottleneck-instance weights often change dynamically or adaptively, and thus the weighted SVM solutions must be repeatedly computed. In this paper, we develop an algorithm that can efficiently and exactly update the weighted SVM solutions for arbitrary change of instance weights. Technically, this contribution can be regarded as an extension of the conventional solution-path algorithm for a single regularization parameter to multiple instance-weight parameters. However, this extension gives rise to a significant problem that breakpoints (at which the solution path turns) have to be identified in high-dimensional space. To facilitate this, we introduce a parametric representation of instance weights. We also provide a geometric interpretation in weight space using a notion of critical region: a polyhedron in which the current affine solution remains to be optimal. Then we find break- Mach Learn (2012) 88:297-330 points at intersections of the solution path and boundaries of polyhedrons. Through extensive experiments on various practical applications, we demonstrate the usefulness of the proposed algorithm.

show abstract

Bi-level path following for cross validated solution of kernel quantile regression

Cited by 8 publications

References 25 publications

The structured elastic net for quantile regression and support vector classification

The structured elastic net for quantile regression and support vector classification

Cross Validation Through Two-Dimensional Solution Surface for Cost-Sensitive SVM

Multi-parametric solution-path algorithm for instance-weighted support vector machines

Contact Info

Product

Resources

About