Algorithms for heavy-tailed statistics: regression, covariance estimation, and beyond

Cherapanamjeri, Yeshwanth; Hopkins, Samuel B.; Kathuria, Tarun; Raghavendra, Prasad; Tripuraneni, Nilesh

doi:10.1145/3357713.3384329

Cited by 23 publications

(17 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lai, Rao, and Vempala similarly use spectral methods for agnostic mean estimation [12]. Hopkins and others achieve similar robustness results for both Gaussian and heavy-tailed distributions, relying on Sum-of-Squares proofs and semi definite programming instead of filtering methods [8,2,9]. This field continues to evolve, with recent works applying gradient estimation and descent to the Sum-of-Squares hierarchy to yield similar error guarantees with faster runtime [1].…”

Section: Related and Prior Workmentioning

confidence: 99%

“…The specific adversarial framework they use, known as γ-corruption, is powerful and can easily be generalized to other models. 2 Definition 2.3 (γ-corruption [4]). Given γ > 0 and a set of samples of size m, the samples are γ-corrupted if an adversary is allowed to inspect the samples, remove m ∼ Bin(γ, m) of them, and replace them with arbitrary points.…”

Section: Robust High-dimensional Mean Estimation By Filteringmentioning

confidence: 99%

See 1 more Smart Citation

Designing Differentially Private Estimators in High Dimensions

Dhar¹,

Huang²

2020

Preprint

View full text Add to dashboard Cite

We study differentially private mean estimation in a high-dimensional setting. Existing differential privacy techniques applied to large dimensions lead to computationally intractable problems or estimators with excessive privacy loss. Recent work in high-dimensional robust statistics has identified computationally tractable mean estimation algorithms with asymptotic dimension-independent error guarantees. We incorporate these results to develop a strict bound on the global sensitivity of the robust mean estimator. This yields a computationally tractable algorithm for differentially private mean estimation in high dimensions with dimensionindependent privacy loss. Finally, we show on synthetic data that our algorithm significantly outperforms classic differential privacy methods, overcoming barriers to high-dimensional differential privacy. * Equal contribution. Author order determined by a random coin flip of a virtual Canadian Loonie.Preprint. Under review.

show abstract

Section: Related and Prior Workmentioning

confidence: 99%

Section: Robust High-dimensional Mean Estimation By Filteringmentioning

confidence: 99%

Designing Differentially Private Estimators in High Dimensions

Dhar¹,

Huang²

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…As sketched in [43, Appendix A], this class contains many popular and powerful frameworks, including local algorithms on graphs, power iteration, and approximate message passing [35,11,56,64,36]. In addition, a recent flurry of work has shown that for many average-case problems in high-dimensional statistics, including planted clique, sparse PCA, community detection, and tensor PCA, low degree polynomials are as powerful as the best polynomial-time algorithms known [55,54,53,7,61,34,22,15,63,70,8,16]. Thus, showing that low degree polynomial algorithms fail at some threshold provides evidence that all polynomial-time algorithms fail at that threshold.…”

Section: Introductionmentioning

confidence: 99%

The Algorithmic Phase Transition of Random $k$-SAT for Low Degree Polynomials

Bresler¹,

Huang²

2021

Preprint

View full text Add to dashboard Cite

Let Φ be a uniformly random k-SAT formula with n variables and m clauses. We study the algorithmic task of finding a satisfying assignment of Φ. It is known that a satisfying assignment exists with high probability at clause density m/n < 2 k log 2 − 1 2 (log 2 + 1) + o k (1), while the best polynomial-time algorithm known, the Fix algorithm of Coja-Oghlan [25], finds a satisfying assignment at the much lower clause density (1 − o k (1))2 k log k/k. This prompts the question: is it possible to efficiently find a satisfying assignment at higher clause densities?To understand the algorithmic threshold of random k-SAT, we study low degree polynomial algorithms, which are a powerful class of algorithms including Fix, Survey Propagation guided decimation (with bounded or mildly growing number of message passing rounds), and paradigms such as message passing and local graph algorithms. We show that low degree polynomial algorithms can find a satisfying assignment at clause density (1 − o k (1))2 k log k/k, matching Fix, and not at clause density (1 + o k (1))κ * 2 k log k/k, where κ * ≈ 4.911. This shows the first sharp (up to constant factor) computational phase transition of random k-SAT for a class of algorithms. Our proof establishes and leverages a new many-way overlap gap property tailored to random k-SAT.

show abstract

“…) with probability 1 − δ, and they need a number of sample of order N log(1/δ)d. The article [11] has been the first to construct a polynomial-time method achieving the rate of the OLS in the Gaussian setting ( f ) − (f * ) ≤ O( log(1/δ)∨d N ). To the date, it is the only procedure running in polynomial algorithm achieving the optimal subgaussian rate.…”

Section: Introductionmentioning

confidence: 99%

“…The goal is to design a procedure which comes with actual working code that attains the optimal sub-gaussian error bound even though the data have only finite moments (up to L4) and in the presence of possibly adversarial outliers. A polynomial-time solution to this problem has been recently discovered [11] but has high runtime due to its use of Sum-of-Square hierarchy programming. At the core of our algorithm is an adaptation of the spectral method introduced in [35] for the mean estimation problem to the linear regression problem.…”

mentioning

confidence: 99%

A spectral algorithm for robust regression with subgaussian rates

Depersin¹

2020

Preprint

View full text Add to dashboard Cite

We study a new linear up to quadratic time algorithm for linear regression in the absence of strong assumptions on the underlying distributions of samples, and in the presence of outliers. The goal is to design a procedure which comes with actual working code that attains the optimal sub-gaussian error bound even though the data have only finite moments (up to L4) and in the presence of possibly adversarial outliers. A polynomial-time solution to this problem has been recently discovered [11] but has high runtime due to its use of Sum-of-Square hierarchy programming. At the core of our algorithm is an adaptation of the spectral method introduced in [35] for the mean estimation problem to the linear regression problem. As a by-product we established a connection between the linear regression problem and the furthest hyperplane problem. From a stochastic point of view, in addition to the study of the classical quadratic and multiplier processes we introduce a third empirical process that comes naturally in the study of the statistical properties of the algorithm. We provide an analysis of this latter process using results from [13].

show abstract

Algorithms for heavy-tailed statistics: regression, covariance estimation, and beyond

Cited by 23 publications

References 8 publications

Designing Differentially Private Estimators in High Dimensions

Designing Differentially Private Estimators in High Dimensions

The Algorithmic Phase Transition of Random $k$-SAT for Low Degree Polynomials

A spectral algorithm for robust regression with subgaussian rates

Contact Info

Product

Resources

About