Regularization, sparse recovery, and median-of-means tournaments

Lugosi, Gábor; Mendelson, Shahar

doi:10.3150/18-bej1046

Cited by 38 publications

(44 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A growing body of recent work has addressed the problem of constructing regression function estimators that work well even when some of the f (X) and Y may be heavy tailed, see Audibert and Catoni [4], Brownlees, Joly, and Lugosi [11], Catoni and Giulini [15]. Chichignoud [59,58], Mendelson [62], and Minsker [65].…”

Section: Median-of-means Tournaments In Regression Problemsmentioning

confidence: 99%

Mean Estimation and Regression Under Heavy-Tailed Distributions: A Survey

2019

Self Cite

View full text Add to dashboard Cite

We survey some of the recent advances in mean estimation and regression function estimation. In particular, we describe sub-Gaussian mean estimators for possibly heavy-tailed data both in the univariate and multivariate settings. We focus on estimators based on median-of-means techniques but other methods such as the trimmed mean and Catoni's estimator are also reviewed. We give detailed proofs for the cornerstone results. We dedicate a section on statistical learning problems-in particular, regression function estimation-in the presence of possibly heavy-tailed data.

show abstract

Section: Median-of-means Tournaments In Regression Problemsmentioning

confidence: 99%

Mean Estimation and Regression Under Heavy-Tailed Distributions: A Survey

2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this article, we address this question by considering an alternative to M-estimators, called median-of-means. Several estimators based on this principle have recently been proposed in the literature [56,42,47,48,49,53,43]. To our knowledge, these articles use the small ball hypothesis [41,54] to treat problems of least square regression or Lipschitzian loss regression.…”

Section: Introductionmentioning

confidence: 99%

Robust classification via MOM minimization

Lecué

Lerasle

Mathieu³

2020

Mach Learn

View full text Add to dashboard Cite

We present an extension of Vapnik's classical empirical risk minimizer (ERM) where the empirical risk is replaced by a median-of-means (MOM) estimator, the new estimators are called MOM minimizers. While ERM is sensitive to corruption of the dataset for many classical loss functions used in classification, we show that MOM minimizers behave well in theory, in the sense that it achieves Vapnik's (slow) rates of convergence under weak assumptions: data are only required to have a finite second moment and some outliers may also have corrupted the dataset.We propose an algorithm inspired by MOM minimizers. These algorithms can be analyzed using arguments quite similar to those used for Stochastic Block Gradient descent. As a proof of concept, we show how to modify a proof of consistency for a descent algorithm to prove consistency of its MOM version. As MOM algorithms perform a smart subsampling, our procedure can also help to reduce substantially time computations and memory ressources when applied to non linear algorithms. These empirical performances are illustrated on both simulated and real datasets.

show abstract

“…It follows from Lemma 6 that there exists α ≥ 1 and f 0 ∈ F such that P L f 0 =r 2 2 (γ) and f − f * = α(f 0 − f * ). According to (30), we have for every k ∈ {1, . .…”

Section: A4 Proof Of Theoremmentioning

confidence: 99%

“…This constraint can be relaxed by considering alternative estimators based on the "median-of-means" (MOM) principle of [37,9,18,1] and the minmax procedure of [3,5]. The resulting minmax MOM estimators have been introduced in [24] for least-squares regression as an alternative to other MOM based procedures [29,30,31,23]. In the case of convex and Lipschitz loss functions, these estimators satisfy the following properties 1) as the ERM, they are efficient under weak assumptions on the noise 2) they achieve optimal rates of convergence under weak stochastic assumptions on the design and 3) the rates are not downgraded by the presence of some outliers in the dataset.These improvements of MOM estimators upon ERM are not surprising.…”

mentioning

confidence: 99%

“…For univariate mean estimation, rate optimal sub-Gaussian deviation bounds can be shown under minimal L 2 moment assumptions for MOM estimators [14] while the empirical mean needs each data to have sub-Gaussian tails to achieve such bounds [13]. In least-squares regression, MOM-based estimators [29,30,31,23,24] inherit these properties, whereas the ERM has downgraded statistical properties under moment assumptions (see Proposition 1.5 in [25]). Furthermore, MOM procedures are resistant to outliers: results hold in the "O ∪ I" framework of [23,24], where inliers or informative data (indexed by I) only satisfy weak moments assumptions and the dataset may contain outliers (indexed by O) on which no assumption is made, see Section 3.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Robust statistical learning with Lipschitz and convex loss functions

Chinot¹,

Lecué²,

Lerasle³

2019

Probab. Theory Relat. Fields

View full text Add to dashboard Cite

We obtain estimation and excess risk bounds for Empirical Risk Minimizers (ERM) and minmax Median-Of-Means (MOM) estimators based on loss functions that are both Lipschitz and convex. Results for the ERM are derived under weak assumptions on the outputs and subgaussian assumptions on the design as in [2]. The difference with [2] is that the global Bernstein condition of this paper is relaxed here into a local assumption. We also obtain estimation and excess risk bounds for minmax MOM estimators under similar assumptions on the output and only moment assumptions on the design. Moreover, the dataset may also contains outliers in both inputs and outputs variables without deteriorating the performance of the minmax MOM estimators.Unlike alternatives based on MOM's principle [24,29], the analysis of minmax MOM estimators is not based on the small ball assumption (SBA) of [22]. In particular, the basic example of non parametric statistics where the learning class is the linear span of localized bases, that does not satisfy SBA [39] can now be handled. Finally, minmax MOM estimators are analysed in a setting where the local Bernstein condition is also dropped out. It is shown to achieve excess risk bounds with exponentially large probability under minimal assumptions insuring only the existence of all objects. * • The hinge loss defined, for any u ∈Ȳ = R and y ∈ Y = {−1, 1}, by¯ (u, y) = max(1 − uy, 0) satisfies Assumption 1 with L = 1.• The Huber loss defined, for anysatisfies Assumption 1 with L = δ.• The quantile loss is defined, for any τ. It satisfies Assumption 1 with L = 1. For τ = 1/2, the quantile loss is the L 1 loss.All along the paper, the following assumption is also granted.Assumption 2. The class F is convex.The empirical risk minimizers (ERM) [43] obtained by minimizing f ∈ F → R N (f ) are expected to be close to the oracle f * . This procedure and its regularized versions have been extensively studied in learning theory [20]. When the loss is both convex and Lipschitz, results have been obtained in practice [4,12] and theory [42]. Risk bounds with exponential deviation inequalities for the ERM can be obtained under weak assumptions on the outputs Y , but stronger assumptions on the design X. Moreover, fast rates of convergence [41] can only be obtained under margin type assumptions such as the Bernstein condition [8,42].The Lipschitz assumption and global Bernstein conditions (that hold over the entire F as in [2]) imply boundedness in L 2 -norm of the class F , see the discussion preceding Assumption 4 for details. This boundedness is not satisfied in linear regression with unbounded design so the results of [2] don't apply to this basic example such as linear regression with a Gaussian design. To bypass this restriction, the global condition is relaxed into a "local" one as in [15,42], see Assumption 4 below.The main constraint in our results on ERM is the assumption on the design. This constraint can be relaxed by considering alternative estimators based on the "median-of-means" (MOM) principle of [...

show abstract

Regularization, sparse recovery, and median-of-means tournaments

Cited by 38 publications

References 16 publications

Mean Estimation and Regression Under Heavy-Tailed Distributions: A Survey

Mean Estimation and Regression Under Heavy-Tailed Distributions: A Survey

Robust classification via MOM minimization

Robust statistical learning with Lipschitz and convex loss functions

Contact Info

Product

Resources

About