Statistical limits of dictionary learning: random matrix theory and the spectral replica method

Barbier, Jean; Macris, Nicolas

doi:10.48550/arxiv.2109.06610

Cited by 6 publications

(20 citation statements)

References 92 publications

(190 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, another very recent work, [BM21], came out during the completion of the present paper. This work develops a "spectral replica" calculation for extensive-rank matrix factorization with non-Gaussian prior and (only) Gaussian channel.…”

Section: Related Workmentioning

confidence: 87%

See 1 more Smart Citation

Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising

Maillard,

Krzakala,

Mézard

et al. 2021

Preprint

View full text Add to dashboard Cite

Factorization of matrices where the rank of the two factors diverges linearly with their sizes has many applications in diverse areas such as unsupervised representation learning, dictionary learning or sparse coding. We consider a setting where the two factors are generated from known componentwise independent prior distributions, and the statistician observes a (possibly noisy) componentwise function of their matrix product. In the limit where the dimensions of the matrices tend to infinity, but their ratios remain fixed, we expect to be able to derive closed form expressions for the optimal mean squared error on the estimation of the two factors. However, this remains a very involved mathematical and algorithmic problem. A related, but simpler, problem is extensive-rank matrix denoising, where one aims to reconstruct a matrix with extensive but usually small rank from noisy measurements. In this paper, we approach both these problems using high-temperature expansions at fixed order parameters. This allows to clarify how previous attempts at solving these problems failed at finding an asymptotically exact solution. We provide a systematic way to derive the corrections to these existing approximations, taking into account the structure of correlations particular to the problem. Finally, we illustrate our approach in detail on the case of extensive-rank matrix denoising. We compare our results with known optimal rotationally-invariant estimators, and show how exact asymptotic calculations of the minimal error can be performed using extensiverank matrix integrals.

show abstract

Section: Related Workmentioning

confidence: 87%

“…This hint is strengthened by recent results in[BM21] using the spectral replica method: in the special case of Gaussian channels Pout, it analytically computes the asymptotic mean free energy with the replica method, and its results strongly suggest that the proper order parameter is indeed a probability measure of eigenvalues.…”

mentioning

confidence: 86%

Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising

Maillard,

Krzakala,

Mézard

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Then, the ℓBNN problem is closely related to the random-design linear-rank matrix inference task, which is known to be challenging to analyze [34]- [36]. We direct the interested reader to recent works by Barbier and Macris [35] and by Maillard et al [36] on this problem, and defer more detailed analysis to future work.…”

Section: N N D → ∞ and P Fixedmentioning

confidence: 99%

Depth induces scale-averaging in overparameterized linear Bayesian neural networks

Zavatone-Veth,

Pehlevan

2021

Preprint

View full text Add to dashboard Cite

Inference in deep Bayesian neural networks is only fully understood in the infinite-width limit, where the posterior flexibility afforded by increased depth washes out and the posterior predictive collapses to a shallow Gaussian process. Here, we interpret finite deep linear Bayesian neural networks as datadependent scale mixtures of Gaussian process predictors across output channels. We leverage this observation to study representation learning in these networks, allowing us to connect limiting results obtained in previous studies within a unified framework. In total, these results advance our analytical understanding of how depth affects inference in a simple class of Bayesian neural networks.

show abstract

“…Linear models always carry information about the model parameters. In general, the overlap is a decreasing function of the load R: For smaller R the problem is better determined, and hence the learned labels are more correlated to the true symbols 17 . This is shown in Fig.…”

Section: A Classical Case: Linear Generative Modelmentioning

confidence: 99%

Bayesian Inference with Nonlinear Generative Models: Comments on Secure Learning

Bereyhi¹,

Loureiro²,

Krząkała³

et al. 2022

Preprint

View full text Add to dashboard Cite

Unlike the classical linear model, nonlinear generative models have been addressed sparsely in the literature. This work aims to bring attention to these models and their secrecy potential. To this end, we invoke the replica method to derive the asymptotic normalized cross entropy in an inverse probability problem whose generative model is described by a Gaussian random field with a generic covariance function. Our derivations further demonstrate the asymptotic statistical decoupling of Bayesian inference algorithms and specify the decoupled setting for a given nonlinear model.The replica solution depicts that strictly nonlinear models establish an all-or-nothing phase transition: There exists a critical load at which the optimal Bayesian inference changes from being perfect to an uncorrelated learning. This finding leads to design of a new secure coding scheme which achieves the secrecy capacity of the wiretap channel. The proposed coding has a significantly smaller codebook size compared to the random coding scheme of Wyner. This interesting result implies that strictly nonlinear generative models are perfectly secured without any secure coding. We justify this latter statement through the analysis of an illustrative model for perfectly secure and reliable inference.

show abstract

Statistical limits of dictionary learning: random matrix theory and the spectral replica method

Cited by 6 publications

References 92 publications

Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising

Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising

Depth induces scale-averaging in overparameterized linear Bayesian neural networks

Bayesian Inference with Nonlinear Generative Models: Comments on Secure Learning

Contact Info

Product

Resources

About