Constrained low-rank matrix estimation: phase transitions, approximate message passing and applications

Abstract

This article is an extended version of previous work of the authors [1, 2] on low-rank matrix estimation in the presence of constraints on the factors into which the matrix is factorized. Lowrank matrix factorization is one of the basic methods used in data analysis for unsupervised learning of relevant features and other types of dimensionality reduction. We present a framework to study the constrained low-rank matrix estimation for a general prior on the factors, and a general output channel through which the matrix is observed. We draw a paralel with the study of vector-spin glass models -presenting a unifying way to study a number of problems considered previously in separate statistical physics works. We present a number of applications for the problem in data analysis. We derive in detail a general form of the low-rank approximate message passing (Low-RAMP) algorithm, that is known in statistical physics as the TAP equations. We thus unify the derivation of the TAP equations for models as different as the Sherrington-Kirkpatrick model, the restricted Boltzmann machine, the Hopfield model or vector (xy, Heisenberg and other) spin glasses. The state evolution of the Low-RAMP algorithm is also derived, and is equivalent to the replica symmetric solution for the large class of vector-spin glass models. In the section devoted to result we study in detail phase diagrams and phase transitions for the Bayes-optimal inference in low-rank matrix estimation. We present a typology of phase transitions and their relation to performance of algorithms such as the Low-RAMP or commonly used spectral methods.

The Geometry of Community Detection via the MMSE Matrix

“…Remark 2. Similar to previous work [3][4][5][6][7][8], our proofs of Theorems 1 and 2 use a channel universality argument to relate the community detection problem to a low-rank estimation problem. Assumption 2 is needed for the proof of Theorem 1, which leverages [5,Theorem 12].…”

“…Inspired by the work of Decelle et al [2], a recent line of work has studied the information-theoretic limits of recovery when the distribution of (X, G) is known. Most of this work has focused on either the two-community SBM [3][4][5][6][7][8][9] or the so-called k-community symmetric SBM [7,[10][11][12]. In all of these cases, performance is summarized in terms of a single numerical value, which is often referred to as the effective signal-to-noise ratio of the problem.…”

Marvels and Pitfalls of the Langevin Algorithm in Noisy High-Dimensional Inference

“…In this section we present the performance of the Bayes-optimal estimator and of the approximate message passing algorithm. This theory is based on a straightforward adaptation of analogous results known for the pure spiked matrix model [9,29,36] and for the pure spiked tensor model [10,17].…”

Mutual Information in Community Detection with Covariate Information and Correlated Networks

“…The main results of this paper apply to the so-called dense network setting where the expected degree d of each node in the network increases with the problem dimension n. In this setting, previous work has provided bounds on the asymptotic minimum mean-squared error of estimating the community labels [3][4][5][6][7][8][9]13]. The analysis in this paper builds upon the recent work in [13], which shows that the mutual information and MMSE in a degree balanced SBM can be characterized in terms of a matrix of effective signal-to-noise ratios.…”

“…A recent line of work has studied the information-theoretic limits of recovery. Most of this work has focused on either the two-community SBM [2][3][4][5][6][7][8][9] or the so-called k-community symmetric SBM [7,[10][11][12]. In all of these cases, performance is summarized in terms of a single numerical value, which is often referred to as the effective signal-to-noise ratio of the problem.…”