Beyond Least-Squares: Fast Rates for Regularized Empirical Risk Minimization through Self-Concordance

Abstract

We consider learning methods based on the regularization of a convex empirical risk by a squared Hilbertian norm, a setting that includes linear predictors and non-linear predictors through positive-definite kernels. In order to go beyond the generic analysis leading to convergence rates of the excess risk as O(1/ √ n) from n observations, we assume that the individual losses are self-concordant, that is, their third-order derivatives are bounded by their secondorder derivatives. This setting includes least-squares, as well as all generalized linear models such as logistic and softmax regression. For this class of losses, we provide a bias-variance decomposition and show that the assumptions commonly made in least-squares regression, such as the source and capacity conditions, can be adapted to obtain fast non-asymptotic rates of convergence by improving the bias terms, the variance terms or both.

Self-Concordant Analysis of Frank-Wolfe Algorithms

“…Statistical analysis using (generalized) self-concordant (SC) functions as a loss function is gaining increasing attention in the machine learning community [1,23,30,31]. This class of loss functions allows to obtain faster statistical rates akin to least-squares.…”

Efficient improper learning for online logistic regression

“…In this paper, we introduce a new practical improper algorithm, that we call AIOLI (Algorithmic efficient Improper Online LogIstic regression), for online logistic regression. The latter is based on Follow The Regularized Leader (FTRL) [McMahan, 2011] with surrogate losses. AIOLI takes inspiration from the Azoury-Warmuth-Vovk forecaster (also named non-linear Ridge regression or AWV) from [Vovk, 2001] and [Azoury and Warmuth, 2001] which adds a non-proper penalty based on the next input x t and from Online Newton Step [Hazan et al, 2007] which leverage the exp-concavity of logistic regression to achieve logarithmic regret.…”

“…Note that the effective dimension is always upper-bounded by d eff (λ) n/λ, providing in the worst case, the regret upperbound of order O(B √ n) for well-chosen λ. Under the capacity condition, which is a classical assumption for kernels (see [Marteau-Ferey et al, 2019] for instance), better bounds on the effective dimension are provided which yield to faster regret rates.…”

“…Apart from [Foster et al, 2018], which is non-practical and also based on an online-to-batch conversion, we are only aware of the works of [Mourtada and Gaïffas, 2019] and [Marteau-Ferey et al, 2019] that improve the exponential constant O(e B ) in the statistical setting. [Marteau-Ferey et al, 2019] make additional assumptions on the data distribution (self-concordance, well-specified model, capacity and source conditions). Their framework is hardly comparable to ours with constants that may be arbitrarily large in our setting.…”

“…data only. [Marteau-Ferey et al, 2019] considered the classical regularized empirical risk minimizer (ERM). Though the latter is proper, using generalized self-concordance properties they could avoid the exponential constant in B under additional assumptions including a well-specified problem, capacity and source conditions.…”