Computational Lower Bounds for Sparse PCA

Berthet, Quentin; Rigollet, Philippe

doi:10.48550/arxiv.1304.0828

Cited by 26 publications

(41 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many problems in high-dimensional statistics are believed to exhibit gaps between what can be achieved information-theoretically (or statistically, i.e., with unbounded computational power) and what is possible with bounded computational power (e.g., in polynomial time). Examples include finding planted cliques [Jer92, DM15b, MPW15, BHK + 19] or dense communities [DKMZ11b,DKMZ11a,HS17] in random graphs, extracting variously structured principal components of random matrices [BR13,LKZ15a,LKZ15b] or tensors [HSS15, HKP + 17], and solving or refuting random constraint satisfaction problems [ACO08,KMOW17].…”

Section: Overviewmentioning

confidence: 99%

Notes on Computational Hardness of Hypothesis Testing: Predictions using the Low-Degree Likelihood Ratio

Kunisky¹,

Wein²,

Bandeira³

2019

Preprint

View full text Add to dashboard Cite

These notes survey and explore an emerging method, which we call the low-degree method, for predicting and understanding statistical-versus-computational tradeoffs in high-dimensional inference problems. In short, the method posits that a certain quantity -the second moment of the low-degree likelihood ratio -gives insight into how much computational time is required to solve a given hypothesis testing problem, which can in turn be used to predict the computational hardness of a variety of statistical inference tasks. While this method originated in the study of the sum-of-squares (SoS) hierarchy of convex programs, we present a self-contained introduction that does not require knowledge of SoS. In addition to showing how to carry out predictions using the method, we include a discussion investigating both rigorous and conjectural consequences of these predictions.These notes include some new results, simplified proofs, and refined conjectures. For instance, we point out a formal connection between spectral methods and the low-degree likelihood ratio, and we give a sharp low-degree lower bound against subexponential-time algorithms for tensor PCA.

show abstract

Section: Overviewmentioning

confidence: 99%

Notes on Computational Hardness of Hypothesis Testing: Predictions using the Low-Degree Likelihood Ratio

Kunisky¹,

Wein²,

Bandeira³

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…They show that this algorithm correctly recovers the support with high probability for sparse parameter k within order √ M , with M being the number of samples. This sample complexity, combining with the lower bounds results in [6,32], suggest that no polynomial time algorithm can do significantly better under their statistical assumptions. There are also a series of papers [42,10,45,11,30] that provide the minimax rate of estimation for sparse PCA.…”

Section: Literature Reviewmentioning

confidence: 72%

Solving sparse principal component analysis with global support

Dey¹,

Molinaro²,

Wang³

2020

Preprint

View full text Add to dashboard Cite

Row-sparse principal component analysis (rsPCA), also known as principal component analysis (PCA) with global support, is the problem of finding the top-r leading principal components such that all these principal components are linear combination of a subset of k variables. rsPCA is a popular dimension reduction tool in statistics that enhances interpretability compared to regular principal component analysis (PCA). Popular methods for solving rsPCA mentioned in literature are either greedy heuristics (in the special case of r = 1) where guarantees on the quality of solution found can be verified under restrictive statistical-models, or algorithms with stationary point convergence guarantee for some regularized reformulation of rsPCA. There are no known good heuristics when r > 1, and more importantly none of the existing computational methods can efficiently verify the quality of the solutions via comparing objective values of feasible solutions with dual bounds, especially in a statistical-model-free setting.We propose: (a) a convex integer programming relaxation of rsPCA that gives upper (dual) bounds for rsPCA, and; (b) a new local search algorithm for finding primal feasible solutions for rsPCA in the general case where r > 1. We also show that, in the worst-case, the dual bounds provided by the convex IP is within an affine function of the global optimal value. Numerical results are reported to demonstrate the advantages of our method.

show abstract

“…In summary, only computationally intractable algorithms are known to reach the statistical limit N = Ω(s) for Sparse PCA, while polynomial time methods are only sub-optimal requiring N = Ω(s 2 ). The study of this computational-to-statistical gap was initiated by [11] who investigated the detection problem via a reduction to the planted clique problem which is conjectured to be computationally hard.…”

Section: Related Workmentioning

confidence: 99%

“…13 12 (15), (17) and the assumption on the noise (11) in the first inequality and 2 d d 12 w ≤ K 2 x 2 with d ≥ 2 in the last one. We conclude that if f is differentiable at x ∈ B(0, x /16π) then x, ṽx < 0.…”

Section: A3 Proof Of Theoremmentioning

confidence: 99%

“…With a sparsity prior, the information-theoretically optimal sample complexity is proportional to the sparsity level s of the signal, on the other hand the best known algorithms (convex methods [28,38,47], iterative thresholding [15,58,62], etc.) require a sample complexity proportional to s 2 for stable recovery, a barrier which might not be resolvable by polynomial-time algorithms [11,10]. Under the generative prior (3), [26] has shown that, compressive phase retrieval is possible via gradient descent over a nonlinear objective with sample complexity proportional (up to log factors) to the underlying signal dimensionality, k. This result suggests that it may be possible to use generative priors to close other computational-to-statistical gaps such as for models (1) and (2).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Nonasymptotic Guarantees for Spiked Matrix Recovery with Generative Priors

Cocola¹,

Hand²,

Voroninski³

2020

Preprint

View full text Add to dashboard Cite

Many problems in statistics and machine learning require the reconstruction of a lowrank signal matrix from noisy data. Enforcing additional prior information on the low-rank component is often key to guaranteeing good recovery performance. One such prior on the low-rank component is sparsity, giving rise to the sparse principal component analysis problem.Unfortunately, this problem suffers from a computational-to-statistical gap, which may be fundamental. In this work, we study an alternative prior where the low-rank component is in the range of a trained generative network. We provide a non-asymptotic analysis with optimal sample complexity, up to logarithmic factors, for low-rank matrix recovery under an expansive-Gaussian network prior. Specifically, we establish a favorable global optimization landscape for a mean squared error optimization, provided the number of samples is on the order of the dimensionality of the input to the generative model. As a result, we establish that generative priors have no computational-to-statistical gap for structured low-rank matrix recovery in the finite data, nonasymptotic regime. We present this analysis in the case of both the Wishart and Wigner spiked matrix models.

show abstract

Computational Lower Bounds for Sparse PCA

Cited by 26 publications

References 16 publications

Notes on Computational Hardness of Hypothesis Testing: Predictions using the Low-Degree Likelihood Ratio

Notes on Computational Hardness of Hypothesis Testing: Predictions using the Low-Degree Likelihood Ratio

Solving sparse principal component analysis with global support

Nonasymptotic Guarantees for Spiked Matrix Recovery with Generative Priors

Contact Info

Product

Resources

About