Asymptotic Analysis of Penalized Likelihood and Related Estimators

Cox, Dennis D.; O’Sullivan, Finbarr

doi:10.1214/aos/1176347872

Cited by 159 publications

(105 citation statements)

References 9 publications

Supporting

Mentioning

104

Contrasting

Order By: Relevance

“…The algorithms under consideration approximately solve a linear programming problem, but tend to perform sub-optimally on noisy data. From the margin bound (43), this is indeed not surprising. The minimum on the right hand side of (43) is not necessarily achieved with the maximum (hard) margin (largest θ).…”

Section: Optimization Of the Marginsmentioning

confidence: 85%

See 1 more Smart Citation

An Introduction to Boosting and Leveraging

Meir

Rätsch

2003

Advanced Lectures on Machine Learning

281

186

View full text Add to dashboard Cite

Abstract. We provide an introduction to theoretical and practical aspects of Boosting and Ensemble learning, providing a useful reference for researchers in the field of Boosting as well as for those seeking to enter this fascinating area of research. We begin with a short background concerning the necessary learning theoretical foundations of weak learners and their linear combinations. We then point out the useful connection between Boosting and the Theory of Optimization, which facilitates the understanding of Boosting and later on enables us to move on to new Boosting algorithms, applicable to a broad spectrum of problems. In order to increase the relevance of the paper to practitioners, we have added remarks, pseudo code, "tricks of the trade", and algorithmic considerations where appropriate. Finally, we illustrate the usefulness of Boosting algorithms by giving an overview of some existing applications. The main ideas are illustrated on the problem of binary classification, although several extensions are discussed.

show abstract

Section: Optimization Of the Marginsmentioning

confidence: 85%

“…However when using Boosting procedures on noisy realworld data, it turns out that regularization (e.g. [103,186,143,43]) is mandatory if overfitting is to be avoided (cf. Section 6).…”

Section: Learning From Data and The Pac Propertymentioning

confidence: 99%

An Introduction to Boosting and Leveraging

Meir

Rätsch

2003

Advanced Lectures on Machine Learning

281

186

View full text Add to dashboard Cite

show abstract

“…This fundamental property goes under the name of representer theorem. The result, which goes back to (Kimeldorf and Wahba 1971) for squared loss functions, extends to differentiable loss functions, see (Cox and O'Sullivan 1990) and (Poggio and Girosi 1992). More recently, it has been shown that the representer theorem holds in a very general setting ).…”

mentioning

confidence: 81%

An algebraic characterization of the optimum of regularized kernel methods

Dinuzzo

Nicolao

2009

Mach Learn

View full text Add to dashboard Cite

The representer theorem for kernel methods states that the solution of the associated variational problem can be expressed as the linear combination of a finite number of kernel functions. However, for non-smooth loss functions, the analytic characterization of the coefficients poses nontrivial problems. Standard approaches resort to constrained optimization reformulations which, in general, lack a closed-form solution. Herein, by a proper change of variable, it is shown that, for any convex loss function, the coefficients satisfy a system of algebraic equations in a fixed-point form, which may be directly obtained from the primal formulation. The algebraic characterization is specialized to regression and classification methods and the fixed-point equations are explicitly characterized for many loss functions of practical interest. The consequences of the main result are then investigated along two directions. First, the existence of an unconstrained smooth reformulation of the original non-smooth problem is proven. Second, in the context of SURE (Stein's Unbiased Risk Estimation), a general formula for the degrees of freedom of kernel regression methods is derived.

show abstract

“…Examples include penalized least square regression, penalized logistic regression, penalized density estimation, and regularization procedures used in more general nonlinear inverse problems. Cox and O'Sullivan (1990) provided a general framework for studying regularization methods. The method of regularization has two components: a data fit functional component and a regularization penalty component.…”

Section: Support Vector Machines For the Nonstandard Situationmentioning

confidence: 99%

Untitled

2002

View full text Add to dashboard Cite

Abstract. The majority of classification algorithms are developed for the standard situation in which it is assumed that the examples in the training set come from the same distribution as that of the target population, and that the cost of misclassification into different classes are the same. However, these assumptions are often violated in real world settings. For some classification methods, this can often be taken care of simply with a change of threshold; for others, additional effort is required. In this paper, we explain why the standard support vector machine is not suitable for the nonstandard situation, and introduce a simple procedure for adapting the support vector machine methodology to the nonstandard situation. Theoretical justification for the procedure is provided. Simulation study illustrates that the modified support vector machine significantly improves upon the standard support vector machine in the nonstandard situation. The computational load of the proposed procedure is the same as that of the standard support vector machine. The procedure reduces to the standard support vector machine in the standard situation.

show abstract

Asymptotic Analysis of Penalized Likelihood and Related Estimators

Cited by 159 publications

References 9 publications

An Introduction to Boosting and Leveraging

An Introduction to Boosting and Leveraging

An algebraic characterization of the optimum of regularized kernel methods

Untitled

Contact Info

Product

Resources

About