Information-theoretic bounds and phase transitions in clustering, sparse PCA, and submatrix localization

Banks, Jess; Moore, Cristopher; Vershynin, Roman; Verzélen, Nicolas; Xu, Jiaming

doi:10.1109/isit.2017.8006706

Cited by 16 publications

(41 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While bounds on the second moment do not a priori imply anything about the recovery problem, it follows from results of Banks et al (2017) that many of our non-detection results imply the corresponding non-recovery results. The value of the second moment also yields bounds on hypothesis testing power (see Proposition 2.5).…”

mentioning

confidence: 61%

“…There will be times when the above second moment is unbounded but we are still able to prove contiguity using a modified second moment that conditions away from rare 'bad' events that would otherwise dominate the second moment. This idea has appeared previously (Arias-Castro and Verzelen, 2014;Verzelen and Arias-Castro, 2015;Banks et al, 2016Banks et al, , 2017.…”

mentioning

confidence: 67%

“…Even more recent work (Onatski, Moreira and Hallin, 2014;Dobriban, 2016;Ke, 2016) elaborates on this point in other spiked models. In the Gaussian Wigner model it has been established by Montanari, Reichman and Zeitouni (2015) and Johnstone and Onatski (2015) that detection is impossible below the spectral threshold, and the former used techniques similar to those of the present paper, which are not fundamentally limited to spherically symmetric models; indeed, these techniques were applied to sparse PCA by Banks et al (2017).…”

mentioning

confidence: 90%

See 2 more Smart Citations

Optimality and sub-optimality of PCA I: Spiked random matrix models

Perry¹,

Wein²,

Bandeira³

et al. 2018

Ann. Statist.

112

159

View full text Add to dashboard Cite

A central problem of random matrix theory is to understand the eigenvalues of spiked random matrix models, introduced by Johnstone, in which a prominent eigenvector (or "spike") is planted into a random matrix. These distributions form natural statistical models for principal component analysis (PCA) problems throughout the sciences. Baik, Ben Arous and Péché showed that the spiked Wishart ensemble exhibits a sharp phase transition asymptotically: when the spike strength is above a critical threshold, it is possible to detect the presence of a spike based on the top eigenvalue, and below the threshold the top eigenvalue provides no information. Such results form the basis of our understanding of when PCA can detect a low-rank signal in the presence of noise. However, under structural assumptions on the spike, not all information is necessarily contained in the spectrum. We study the statistical limits of tests for the presence of a spike, including non-spectral tests. Our results leverage Le Cam's notion of contiguity, and include: i) For the Gaussian Wigner ensemble, we show that PCA achieves the optimal detection threshold for certain natural priors for the spike. ii) For any non-Gaussian Wigner ensemble, PCA is sub-optimal for detection. However, an efficient variant of PCA achieves the optimal threshold (for natural priors) by pre-transforming the matrix entries. iii) For the Gaussian Wishart ensemble, the PCA threshold is optimal for positive spikes (for natural priors) but this is not always the case for negative spikes. * The first two authors contributed equally.

show abstract

mentioning

confidence: 61%

mentioning

confidence: 67%

mentioning

confidence: 90%

See 1 more Smart Citation

Optimality and sub-optimality of PCA I: Spiked random matrix models

Perry¹,

Wein²,

Bandeira³

et al. 2018

Ann. Statist.

112

159

View full text Add to dashboard Cite

show abstract

“…The supervised on labeled curve corresponds to (2) with the parameter α replaced by αη and is the best possible performance when only a fraction η of the data points having labels are used. The unsupervised curve corresponds to (3) where all the data points are used but without any label. Finally, the oracle curve corresponds to (1) where the centers of the clusters are known (corresponding to the case α → ∞).…”

Section: Unsupervised Casementioning

confidence: 99%

“…Using exact but non-rigorous methods from statistical physics, [6,24] determine the critical values for α and σ at which it becomes information-theoretically possible to reconstruct the membership into clusters better than chance. Rigorous results on this model are given in [3] where bounds on the critical values are obtained. The precise thresholds were then determined in [27].…”

Section: Related Workmentioning

confidence: 99%

Asymptotic Bayes Risk for Gaussian Mixture in a Semi-Supervised Setting

Lelarge

Leo

2019

2019 IEEE 8th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP)

View full text Add to dashboard Cite

Semi-supervised learning (SSL) uses unlabeled data for training and has been shown to greatly improve performance when compared to a supervised approach on the labeled data available. This claim depends both on the amount of labeled data available and on the algorithm used. In this paper, we compute analytically the gap between the best fully-supervised approach using only labeled data and the best semi-supervised approach using both labeled and unlabeled data. We quantify the best possible increase in performance obtained thanks to the unlabeled data, i.e. we compute the accuracy increase due to the information contained in the unlabeled data. Our work deals with a simple high-dimensional Gaussian mixture model for the data in a Bayesian setting. Our rigorous analysis builds on recent theoretical breakthroughs in high-dimensional inference and a large body of mathematical tools from statistical physics initially developed for spin glasses.

show abstract

Fundamental limits of symmetric low-rank matrix estimation

Lelarge

Leo

2018

Probab. Theory Relat. Fields

138

208

View full text Add to dashboard Cite

We consider the high-dimensional inference problem where the signal is a low-rank symmetric matrix which is corrupted by an additive Gaussian noise. Given a probabilistic model for the low-rank matrix, we compute the limit in the large dimension setting for the mutual information between the signal and the observations, as well as the matrix minimum mean square error, while the rank of the signal remains constant. We also show that our model extends beyond the particular case of additive Gaussian noise and we prove an universality result connecting the community detection problem to our Gaussian framework. We unify and generalize a number of recent works on PCA, sparse PCA, submatrix localization or community detection by computing the information-theoretic limits for these problems in the high noise regime. In addition, we show that the posterior distribution of the signal given the observations is characterized by a parameter of the same dimension as the square of the rank of the signal (i.e. scalar in the case of rank one). This allows to locate precisely the information-theoretic thresholds for the above mentioned problems. Finally, we connect our work with the hard but detectable conjecture in statistical physics. * M.L. and L.M. are with INRIA-ENS,

show abstract

Information-theoretic bounds and phase transitions in clustering, sparse PCA, and submatrix localization

Cited by 16 publications

References 40 publications

Optimality and sub-optimality of PCA I: Spiked random matrix models

Optimality and sub-optimality of PCA I: Spiked random matrix models

Asymptotic Bayes Risk for Gaussian Mixture in a Semi-Supervised Setting

Fundamental limits of symmetric low-rank matrix estimation

Contact Info

Product

Resources

About