Distribution-Free Robust Linear Regression

Mourtada, Jaouad; Vaškevičius, Tomas; Zhivotovskiy, Nikita

doi:10.48550/arxiv.2102.12919

Cited by 3 publications

(3 citation statements)

References 60 publications

(156 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We will use the following classical estimator (Györfi et al, 2002, Thm 11.3); it is quoted here with an improved bound which is proven in (Mourtada et al, 2021 (w ⊤ (x, 1) − f * (x)) 2 dQ(x).…”

Section: A Simplified Version Of Our Estimatormentioning

confidence: 99%

Efficient Minimax Optimal Estimators For Multivariate Convex Regression

Kur¹,

Putterman²

2022

Preprint

View full text Add to dashboard Cite

We study the computational aspects of the task of multivariate convex regression in dimension d ≥ 5. We present the first computationally efficient minimax optimal (up to logarithmic factors) estimators for the tasks of (i) L-Lipschitz convex regression (ii) Γ-bounded convex regression under polytopal support. The proof of the correctness of these estimators uses a variety of tools from different disciplines, among them empirical process theory, stochastic geometry, and potential theory. This work is the first to show the existence of efficient minimax optimal estimators for non-Donsker classes that their corresponding Least Squares Estimators are provably minimax sub-optimal; a result of independent interest.

show abstract

Section: A Simplified Version Of Our Estimatormentioning

confidence: 99%

Efficient Minimax Optimal Estimators For Multivariate Convex Regression

Kur¹,

Putterman²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The paper of Hardt, Recht, and Singer [19] on the stability of gradient descent methods has generated a wave of interest in this direction. Recent works use various notions of stability in their analysis: some authors are motivated by the analysis of gradient descent algorithms [31,29,15,4], while others use the notion of average stability to obtain the in-expectation O(1/n) rate for regularized regression [28,17,46] and some more specific improper learning procedures [36,37]. One of the key open questions left in [19] is related to the lack of high probability generalization bounds.…”

Section: Introductionmentioning

confidence: 99%

Stability and Deviation Optimal Risk Bounds with Convergence Rate $O(1/n)$

Klochkov¹,

Zhivotovskiy²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

The sharpest known high probability generalization bounds for uniformly stable algorithms (Feldman, Vondrák, NeurIPS 2018, COLT, 2019, (Bousquet, Klochkov, Zhivotovskiy, COLT, 2020) contain a generally inevitable sampling error term of order Θ(1/ √ n). When applied to excess risk bounds, this leads to suboptimal results in several standard stochastic convex optimization problems. We show that if the so-called Bernstein condition is satisfied, the term Θ(1/ √ n) can be avoided, and high probability excess risk bounds of order up to O(1/n) are possible via uniform stability. Using this result, we show a high probability excess risk bound with the rate O(log n/n) for strongly convex and Lipschitz losses valid for any empirical risk minimization method. This resolves a question of Shalev-Shwartz, Shamir, Srebro, and Sridharan (COLT, 2009). We discuss how O(log n/n) high probability excess risk bounds are possible for projected gradient descent in the case of strongly convex and Lipschitz losses without the usual smoothness assumption.

show abstract

“…We first observe that an important downside of the small method is the necessity to have Q 2ξ (E, a) > 0, i.e, the random vector a must satisfy a small ball condition, otherwise the conclusion of the theorem is empty, the right hand side becomes negative. The small ball condition has been analysed in a recent line of work dedicated to remove such condition in many different problems in mathematical statistics and mathematical signal processing [14,19,21,25]. Indeed, the estimates provided by the small ball method quickly deteriorates when Q 2ξ (E, a) is not an absolute constant.…”

Section: Introductionmentioning

confidence: 99%

Robust Sparse Recovery with Sparse Bernoulli matrices via Expanders

Abdalla¹

2021

Preprint

View full text Add to dashboard Cite

Sparse binary matrices are of great interest in the field of compressed sensing. This class of matrices make possible to perform signal recovery with lower storage costs and faster decoding algorithms. In particular, random matrices formed by i.i.d Bernoulli p random variables are of practical relevance in the context of nonnegative sparse recovery.In this work, we investigate the robust nullspace property of sparse Bernoulli p matrices. Previous results in the literature establish that such matrices can accurately recover n-dimensional s-sparse vectors with m = O s c(p) log en s measurements, where c(p) ≤ p is a constant that only depends on the parameter p. These results suggest that, when p vanishes, the sparse Bernoulli matrix requires considerably more measurements than the minimal necessary achieved by the standard isotropic subgaussian designs. We show that this is not true. Our main result characterizes, for a wide range of sparsity levels s, the smallest p such that it is possible to perform sparse recovery with the minimal number of measurements. We also provide matching lower bounds to establish the optimality of our results.

show abstract

Distribution-Free Robust Linear Regression

Cited by 3 publications

References 60 publications

Efficient Minimax Optimal Estimators For Multivariate Convex Regression

Efficient Minimax Optimal Estimators For Multivariate Convex Regression

Stability and Deviation Optimal Risk Bounds with Convergence Rate $O(1/n)$

Robust Sparse Recovery with Sparse Bernoulli matrices via Expanders

Contact Info

Product

Resources

About