Information-theoretic bounds and phase transitions in clustering, sparse PCA, and submatrix localization

Banks, Jess; Moore, Cristopher; Verzélen, Nicolas; Vershynin, Roman; Xu, Jiaming

doi:10.48550/arxiv.1607.05222

Cited by 9 publications

(38 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also study sparse Rademacher priors, where we show the spectral threshold is sometimes optimal and sometimes suboptimal depending on the sparsity level (Section 3.7). Our results here are similar to those of and Banks et al [2016b], but extend them by proving non-detection up to the spectral threshold for sufficiently high density.…”

Section: Introductionsupporting

confidence: 90%

“…Even more recent work [Dobriban, 2016, Ke, 2016 elaborates on this point in other spiked models. Similar results were established in the Gaussian Wigner case by Montanari et al [2015], through techniques similar to those of the present paper, which are not fundamentally limited to spherically symmetric models; indeed, these techniques were applied to sparse PCA in Banks et al [2016b].…”

Section: Introductionsupporting

confidence: 79%

“…We will do this through computing a particular second moment, related to the χ 2 -divergence as 1 + χ 2 (P n ||Q n ), through a form of the second moment method: Lemma 2.3 (see e.g. Montanari et al [2015], Banks et al [2016b]). Let {P n } and {Q n } be two sequences of probability measures on (Ω n , F n ).…”

Section: Contiguity and The Second Moment Methodsmentioning

confidence: 99%

“…In general, the detection problem and recovery problem can have different thresholds, but this is due to unnatural counterexamples. In many settings we will be able to obtain nonrecovery results by directly appealing to the following result of Banks et al [2016b]:…”

Section: Contiguity and The Second Moment Methodsmentioning

confidence: 99%

“…While bounds on the second moment do not a priori imply anything about the recovery problem, we appeal to a result from Banks et al [2016b] to make this connection and show that many of our non-detection results translate to non-recovery results as well. In addition, the value of the second moment yields specific bounds on the tradeoff between type I and type II error (see Proposition 2.5), as illustrated in Figures 1 and 2.…”

Section: Introductionmentioning

confidence: 97%

See 4 more Smart Citations

Optimality and sub-optimality of PCA I: Spiked random matrix models

Perry¹,

Wein²,

Bandeira³

et al. 2018

Ann. Statist.

112

159

View full text Add to dashboard Cite

A central problem of random matrix theory is to understand the eigenvalues of spiked random matrix models, introduced by Johnstone, in which a prominent eigenvector (or "spike") is planted into a random matrix. These distributions form natural statistical models for principal component analysis (PCA) problems throughout the sciences. Baik, Ben Arous and Péché showed that the spiked Wishart ensemble exhibits a sharp phase transition asymptotically: when the spike strength is above a critical threshold, it is possible to detect the presence of a spike based on the top eigenvalue, and below the threshold the top eigenvalue provides no information. Such results form the basis of our understanding of when PCA can detect a low-rank signal in the presence of noise. However, under structural assumptions on the spike, not all information is necessarily contained in the spectrum. We study the statistical limits of tests for the presence of a spike, including non-spectral tests. Our results leverage Le Cam's notion of contiguity, and include: i) For the Gaussian Wigner ensemble, we show that PCA achieves the optimal detection threshold for certain natural priors for the spike. ii) For any non-Gaussian Wigner ensemble, PCA is sub-optimal for detection. However, an efficient variant of PCA achieves the optimal threshold (for natural priors) by pre-transforming the matrix entries. iii) For the Gaussian Wishart ensemble, the PCA threshold is optimal for positive spikes (for natural priors) but this is not always the case for negative spikes. * The first two authors contributed equally.

show abstract

Section: Introductionsupporting

confidence: 90%

Section: Introductionsupporting

confidence: 79%

Section: Contiguity and The Second Moment Methodsmentioning

confidence: 99%

Section: Contiguity and The Second Moment Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 97%

See 3 more Smart Citations

Optimality and sub-optimality of PCA I: Spiked random matrix models

Perry¹,

Wein²,

Bandeira³

et al. 2018

Ann. Statist.

112

159

View full text Add to dashboard Cite

show abstract

Fundamental limits of symmetric low-rank matrix estimation

Lelarge

Leo

2018

Probab. Theory Relat. Fields

138

208

View full text Add to dashboard Cite

We consider the high-dimensional inference problem where the signal is a low-rank symmetric matrix which is corrupted by an additive Gaussian noise. Given a probabilistic model for the low-rank matrix, we compute the limit in the large dimension setting for the mutual information between the signal and the observations, as well as the matrix minimum mean square error, while the rank of the signal remains constant. We also show that our model extends beyond the particular case of additive Gaussian noise and we prove an universality result connecting the community detection problem to our Gaussian framework. We unify and generalize a number of recent works on PCA, sparse PCA, submatrix localization or community detection by computing the information-theoretic limits for these problems in the high noise regime. In addition, we show that the posterior distribution of the signal given the observations is characterized by a parameter of the same dimension as the square of the rank of the signal (i.e. scalar in the case of rank one). This allows to locate precisely the information-theoretic thresholds for the above mentioned problems. Finally, we connect our work with the hard but detectable conjecture in statistical physics. * M.L. and L.M. are with INRIA-ENS,

show abstract

Constrained low-rank matrix estimation: phase transitions, approximate message passing and applications

2017

View full text Add to dashboard Cite

This article is an extended version of previous work of the authors [1, 2] on low-rank matrix estimation in the presence of constraints on the factors into which the matrix is factorized. Lowrank matrix factorization is one of the basic methods used in data analysis for unsupervised learning of relevant features and other types of dimensionality reduction. We present a framework to study the constrained low-rank matrix estimation for a general prior on the factors, and a general output channel through which the matrix is observed. We draw a paralel with the study of vector-spin glass models -presenting a unifying way to study a number of problems considered previously in separate statistical physics works. We present a number of applications for the problem in data analysis. We derive in detail a general form of the low-rank approximate message passing (Low-RAMP) algorithm, that is known in statistical physics as the TAP equations. We thus unify the derivation of the TAP equations for models as different as the Sherrington-Kirkpatrick model, the restricted Boltzmann machine, the Hopfield model or vector (xy, Heisenberg and other) spin glasses. The state evolution of the Low-RAMP algorithm is also derived, and is equivalent to the replica symmetric solution for the large class of vector-spin glass models. In the section devoted to result we study in detail phase diagrams and phase transitions for the Bayes-optimal inference in low-rank matrix estimation. We present a typology of phase transitions and their relation to performance of algorithms such as the Low-RAMP or commonly used spectral methods.

show abstract

Information-theoretic bounds and phase transitions in clustering, sparse PCA, and submatrix localization

Cited by 9 publications

References 20 publications

Optimality and sub-optimality of PCA I: Spiked random matrix models

Optimality and sub-optimality of PCA I: Spiked random matrix models

Fundamental limits of symmetric low-rank matrix estimation

Constrained low-rank matrix estimation: phase transitions, approximate message passing and applications

Contact Info

Product

Resources

About