Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising

Abstract

Factorization of matrices where the rank of the two factors diverges linearly with their sizes has many applications in diverse areas such as unsupervised representation learning, dictionary learning or sparse coding. We consider a setting where the two factors are generated from known componentwise independent prior distributions, and the statistician observes a (possibly noisy) componentwise function of their matrix product. In the limit where the dimensions of the matrices tend to infinity, but their ratios remain fixed, we expect to be able to derive closed form expressions for the optimal mean squared error on the estimation of the two factors. However, this remains a very involved mathematical and algorithmic problem. A related, but simpler, problem is extensive-rank matrix denoising, where one aims to reconstruct a matrix with extensive but usually small rank from noisy measurements. In this paper, we approach both these problems using high-temperature expansions at fixed order parameters. This allows to clarify how previous attempts at solving these problems failed at finding an asymptotically exact solution. We provide a systematic way to derive the corrections to these existing approximations, taking into account the structure of correlations particular to the problem. Finally, we illustrate our approach in detail on the case of extensive-rank matrix denoising. We compare our results with known optimal rotationally-invariant estimators, and show how exact asymptotic calculations of the minimal error can be performed using extensiverank matrix integrals.

Depth induces scale-averaging in overparameterized linear Bayesian neural networks

“…The natural setup for this joint asymptotic limit is a random design teacherstudent setting, in which the input examples are independent and identically distributed samples from some distribution and the targets are generated by a linear model with random coefficient matrix. Then, the ℓBNN problem is closely related to the random-design linear-rank matrix inference task, which is known to be challenging to analyze [34]- [36]. We direct the interested reader to recent works by Barbier and Macris [35] and by Maillard et al [36] on this problem, and defer more detailed analysis to future work.…”

“…Then, the ℓBNN problem is closely related to the random-design linear-rank matrix inference task, which is known to be challenging to analyze [34]- [36]. We direct the interested reader to recent works by Barbier and Macris [35] and by Maillard et al [36] on this problem, and defer more detailed analysis to future work.…”

Bayesian Inference with Nonlinear Generative Models: Comments on Secure Learning

“…are recovered wrongly, leading to m ⋆ < 1. 18 Here, σ 2 is small but fixed. However, by setting R small enough, we can send Cm⋆ very close to one.…”

A Concise Tutorial on Approximate Message Passing

“…Further, Park et al [37] developed bilinear GAMP (BiG-AMP) which extends the GAMP algorithm to bilinear model in which both the signal of interest and measurement matrix are unknown. Recent works showed that the BiG-AMP can be obtained by Plefka-Georges-Yedidia method [38], [39]. Following VAMP, [40], [41] developed a generalized linear model VAMP (GLM-VAMP) algorithm by constructing an equivalent linear model.…”