Fundamental limits in structured principal component analysis and how to reach them

Abstract

How do statistical dependencies in measurement noise influence high-dimensional inference? To answer this, we study the paradigmatic spiked matrix model of principal components analysis (PCA), where a rank-one matrix is corrupted by additive noise. We go beyond the usual independence assumption on the noise entries, by drawing the noise from a low-order polynomial orthogonal matrix ensemble. The resulting noise correlations make the setting relevant for applications but analytically challenging. We provide characterization of the Bayes optimal limits of inference in this model. If the spike is rotation invariant, we show that standard spectral PCA is optimal. However, for more general priors, both PCA and the existing approximate message-passing algorithm (AMP) fall short of achieving the information-theoretic limits, which we compute using the replica method from statistical physics. We thus propose an AMP, inspired by the theory of adaptive Thouless–Anderson–Palmer equations, which is empirically observed to saturate the conjectured theoretical limit. This AMP comes with a rigorous state evolution analysis tracking its performance. Although we focus on specific noise distributions, our methodology can be generalized to a wide class of trace matrix ensembles at the cost of more involved expressions. Finally, despite the seemingly strong assumption of rotation-invariant noise, our theory empirically predicts algorithmic performance on real data, pointing at strong universality properties.

Joint modelling of whole genome sequence data for human height via approximate message passing

“…We overcome this issue by developing a new approach for GWAS inference, dubbed genomic Vector Approximate Message Passing (gVAMP). Approximate Message Passing (AMP) [11][12][13] refers to a family of iterative algorithms with several attractive properties: (i) AMP allows the usage of a wide range of Bayesian priors; (ii) the AMP performance for high-dimensional data can be precisely characterized by a simple recursion called state evolution [14]; (iii) using state evolution, joint association test statistics can be obtained [15]; and (iv) AMP achieves Bayes-optimal performance in several settings [15][16][17]. However, we find that existing AMP algorithms proposed for various applications [18][19][20][21] cannot be transferred to biobank analyses as: (i) they are entirely infeasible at scale, requiring expensive singular value decompositions; and (ii) they give diverging estimates of the signal in either simulated genomic data or the UK Biobank data.…”

“…The fundamental limits of inference have been precisely characterized, and it has been shown that efficient algorithms, such as Approximate Message Passing (AMP), can meet such limits [8,9]. Specifically, AMP represents a family of iterative algorithms that has been applied to a range of statistical estimation problems, including 1 linear regression [10,11], generalised linear models [12][13][14], and low-rank matrix estimation [9,15].…”

The decimation scheme for symmetric matrix factorization

“…The spectral properties of low rank perturbations of highrank matrices (such as the Wigner matrix Z) are by now largely understood in random matrix theory, and they can give rise to the celebrated BBP carry out a thorough study of carry out a thorough study of transition [16], further studied and extended in [17][18][19][20][21][22][23][24]. Thanks to the effort of a wide interdisciplinary community, we also have a control on the asymptotic behaviour of the posterior measure (2) and an exact formula for the free entropy associated to the lowrank problem [25][26][27][28][29][30][31][32] (recently extended to rotational invariant noise [33]), which yields the Bayes-optimal limit of the noise allowing the reconstruction of the low-rank spike. Finally, a particular class of algorithms, known as approximate message passing (AMP) [34][35][36][37][38], is able to perform factorization up to this Bayes-optimal limit.…”

Matrix factorization with neural networks

“…The spectral properties of low rank perturbations of high-rank matrices (such as the Wigner matrix Z) are by now largely understood in random matrix theory, and they can give rise to the celebrated BBP carry out a thorough study of carry out a thorough study of transition [16], further studied and extended in [17][18][19][20][21][22][23][24]. Thanks to the effort of a wide interdisciplinary community, we also have a control on the asymptotic behaviour of the posterior measure (2) and an exact formula for the free entropy associated to the low-rank problem [25][26][27][28][29][30][31][32] (recently extended to rotational invariant noise [33]), which yields the Bayes-optimal limit of the noise allowing the reconstruction of the low-rank spike. Finally, a particular class of algorithms, known as Approximate Message Passing (AMP) [34][35][36][37][38], is able to perform factorization up to this Bayes-optimal limit.…”