Estimation bounds and sharp oracle inequalities of regularized procedures with Lipschitz loss functions

Alquier, Pierre; Cottet, Vincent; Lecué, Guillaume

doi:10.48550/arxiv.1702.01402

Cited by 6 publications

(26 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1bit matrix completion was first studied in Davenport et al (2014), where the observed entries are assumed to be sampled uniformly at random. This problem was also studied among others by (Cai and Zhou, 2013;Klopp et al, 2015;Alquier et al, 2017). Matrix completion with exponential family noise (for a single matrix) was previously considered in Lafond (2015) and Gunasekar et al (2014).…”

Section: Introductionmentioning

confidence: 93%

“…). More details for these constants can be found in Propositions 6.1 and 6.3 in Alquier et al (2017). Then, the excess risk with respect to these two losses under the uniform sampling is given by: Corollary 3.…”

Section: General Lossesmentioning

confidence: 99%

“…Remark 2. In 1-bit matrix completion (Davenport et al, 2014;Klopp et al, 2015;Alquier et al, 2017), instead of observing the actual entries of the unknown matrix M ∈ R d×D , for a random subset of its entries Ω we observe {Y ij ∈ {+1, −1} : (i, j) ∈ Ω}, where Y ij = 1 with probability f (M ij ) for some link-function f . In Davenport et al (2014) the parameter M is estimated by minimizing the negative log-likelihood under the constraints M ∞ ≤ γ and M * ≤ γ √ rdD for some r > 0.…”

Section: Estimation Proceduresmentioning

confidence: 99%

“…This assumption has been extensively studied in the learning theory literature (Mendelson, 2008;Zhang, 2004;Bartlett et al, 2004;Alquier et al, 2017;Elsener and van de Geer, 2018), and it is called "Bernstein" condition. It is satisfied in various cases of loss function (Alquier et al, 2017) and it ensures a sufficient convexity of the risk around the oracle defined in (7). Note that when the loss function v is strongly convex, the risk function inherits this property and automatically satisfies the margin condition.…”

Section: General Lossesmentioning

confidence: 99%

“…Note that when the loss function v is strongly convex, the risk function inherits this property and automatically satisfies the margin condition. In other cases, this condition requires strong assumptions on the distribution of the observations, for instance for hinge loss or quantile loss (see Section 6 in Alquier et al (2017)). The following result gives an upper bound on the excess risk of the estimator M.…”

Section: General Lossesmentioning

confidence: 99%

See 4 more Smart Citations

Collective Matrix Completion

Alaya,

Klopp

2018

Preprint

View full text Add to dashboard Cite

Matrix completion aims to reconstruct a data matrix based on observations of a small number of its entries. Usually in matrix completion a single matrix is considered, which can be, for example, a rating matrix in recommendation system. However, in practical situations, data is often obtained from multiple sources which results in a collection of matrices rather than a single one. In this work, we consider the problem of collective matrix completion with multiple and heterogeneous matrices, which can be count, binary, continuous, etc. We first investigate the setting where, for each source, the matrix entries are sampled from an exponential family distribution. Then, we relax the assumption of exponential family distribution for the noise. In this setting, we do not assume any specific model for the observations. The estimation procedures are based on minimizing the sum of a goodness-of-fit term and the nuclear norm penalization of the whole collective matrix. We prove that the proposed estimators achieve fast rates of convergence under the two considered settings and we corroborate our results with numerical experiments.

show abstract

Section: Introductionmentioning

confidence: 93%

Section: General Lossesmentioning

confidence: 99%

Section: Estimation Proceduresmentioning

confidence: 99%

Section: General Lossesmentioning

confidence: 99%

Section: General Lossesmentioning

confidence: 99%

See 3 more Smart Citations

Collective Matrix Completion

Alaya,

Klopp

2018

Preprint

View full text Add to dashboard Cite

show abstract

Robust statistical learning with Lipschitz and convex loss functions

Chinot¹,

Lecué²,

Lerasle³

2019

Probab. Theory Relat. Fields

Self Cite

View full text Add to dashboard Cite

We obtain estimation and excess risk bounds for Empirical Risk Minimizers (ERM) and minmax Median-Of-Means (MOM) estimators based on loss functions that are both Lipschitz and convex. Results for the ERM are derived under weak assumptions on the outputs and subgaussian assumptions on the design as in [2]. The difference with [2] is that the global Bernstein condition of this paper is relaxed here into a local assumption. We also obtain estimation and excess risk bounds for minmax MOM estimators under similar assumptions on the output and only moment assumptions on the design. Moreover, the dataset may also contains outliers in both inputs and outputs variables without deteriorating the performance of the minmax MOM estimators.Unlike alternatives based on MOM's principle [24,29], the analysis of minmax MOM estimators is not based on the small ball assumption (SBA) of [22]. In particular, the basic example of non parametric statistics where the learning class is the linear span of localized bases, that does not satisfy SBA [39] can now be handled. Finally, minmax MOM estimators are analysed in a setting where the local Bernstein condition is also dropped out. It is shown to achieve excess risk bounds with exponentially large probability under minimal assumptions insuring only the existence of all objects. * • The hinge loss defined, for any u ∈Ȳ = R and y ∈ Y = {−1, 1}, by¯ (u, y) = max(1 − uy, 0) satisfies Assumption 1 with L = 1.• The Huber loss defined, for anysatisfies Assumption 1 with L = δ.• The quantile loss is defined, for any τ. It satisfies Assumption 1 with L = 1. For τ = 1/2, the quantile loss is the L 1 loss.All along the paper, the following assumption is also granted.Assumption 2. The class F is convex.The empirical risk minimizers (ERM) [43] obtained by minimizing f ∈ F → R N (f ) are expected to be close to the oracle f * . This procedure and its regularized versions have been extensively studied in learning theory [20]. When the loss is both convex and Lipschitz, results have been obtained in practice [4,12] and theory [42]. Risk bounds with exponential deviation inequalities for the ERM can be obtained under weak assumptions on the outputs Y , but stronger assumptions on the design X. Moreover, fast rates of convergence [41] can only be obtained under margin type assumptions such as the Bernstein condition [8,42].The Lipschitz assumption and global Bernstein conditions (that hold over the entire F as in [2]) imply boundedness in L 2 -norm of the class F , see the discussion preceding Assumption 4 for details. This boundedness is not satisfied in linear regression with unbounded design so the results of [2] don't apply to this basic example such as linear regression with a Gaussian design. To bypass this restriction, the global condition is relaxed into a "local" one as in [15,42], see Assumption 4 below.The main constraint in our results on ERM is the assumption on the design. This constraint can be relaxed by considering alternative estimators based on the "median-of-means" (MOM) principle of [...

show abstract

Excess risk bounds in robust empirical risk minimization

Minsker

Mathieu

2021

Information and Inference: A Journal of the IMA

View full text Add to dashboard Cite

This paper investigates robust versions of the general empirical risk minimization algorithm, one of the core techniques underlying modern statistical methods. Success of the empirical risk minimization is based on the fact that for a ‘well-behaved’ stochastic process $\left \{ f(X), \ f\in \mathscr F\right \}$ indexed by a class of functions $f\in \mathscr F$, averages $\frac{1}{N}\sum _{j=1}^N f(X_j)$ evaluated over a sample $X_1,\ldots ,X_N$ of i.i.d. copies of $X$ provide good approximation to the expectations $\mathbb E f(X)$, uniformly over large classes $f\in \mathscr F$. However, this might no longer be true if the marginal distributions of the process are heavy tailed or if the sample contains outliers. We propose a version of empirical risk minimization based on the idea of replacing sample averages by robust proxies of the expectations and obtain high-confidence bounds for the excess risk of resulting estimators. In particular, we show that the excess risk of robust estimators can converge to $0$ at fast rates with respect to the sample size $N$, referring to the rates faster than $N^{-1/2}$. We discuss implications of the main results to the linear and logistic regression problems and evaluate the numerical performance of proposed methods on simulated and real data.

show abstract

Estimation bounds and sharp oracle inequalities of regularized procedures with Lipschitz loss functions

Cited by 6 publications

References 0 publications

Collective Matrix Completion

Collective Matrix Completion

Robust statistical learning with Lipschitz and convex loss functions

Excess risk bounds in robust empirical risk minimization

Contact Info

Product

Resources

About