Information-Theoretic Limits for the Matrix Tensor Product

Abstract

This paper studies a high-dimensional inference problem involving the matrix tensor product of random matrices. This problem generalizes a number of contemporary data science problems including the spiked matrix models used in sparse principal component analysis and covariance estimation and the stochastic block model used in network analysis. The main results are single-letter formulas (i.e., analytical expressions that can be approximated numerically) for the mutual information and the minimum mean-squared error (MMSE) in the Bayes optimal setting where the distributions of all random quantities are known. We provide non-asymptotic bounds and show that our formulas describe exactly the leading order terms in the mutual information and MMSE in the high-dimensional regime where the number of rows n and number of columns d scale with d = O(n α) for some α < 1/20. On the technical side, this paper introduces some new techniques for the analysis of high-dimensional matrix-valued signals. Specific contributions include a novel extension of the adaptive interpolation method that uses order-preserving positive semidefinite interpolation paths, and a variance inequality between the overlap and the free energy that is based on continuous-time I-MMSE relations.

The Solution of the Deep Boltzmann Machine on the Nishimori Line

“…This ensures the existence of a unique global solution over [0, 1] to the system of ODEs (68)-(69). Moreover, the latter implies also that the map −→ Q (•) is still regular, because F is positive as proved in Lemma 2 and ∂E m e t ∂ Q ,r ≥ 0 thanks again to (28). This guarantees the positivity of the trace in (64) and forces the vanishing of the remainder R in -average by Lemma 4.…”

“…After the completion of this work, paper [28] was brought to our attention where the mutual information for a wide class of inference problems is solved by means of a variational principle. While it is possible to obtain our model as an instance of the one considered there, the variational principle presented has no clear correspondence to ours.…”

“…Proof of (16). Equations ( 27) and (28) imply that the quenched pressure is convex in each h r . Hence it is possible to exchange the derivative w.r.t.…”

“…The perturbation is here introduced as an initial condition in order to have the interpolating functions in the form (44). Notice that f e is constant, while f o is a positive Lipschitz function of Q (t) ∈ (0, ∞) K thanks to identity (28) (where N is fixed). Therefore, by Cauchy-Lipschitz's theorem, the system of ODEs ( 56)-(57) has a unique global solution Q (t) , t ∈ [0, 1] , whose components are positive.…”

“…The authors thank Jean Barbier and Francesco Guerra for interesting discussions. We acknowledge Jean-Cristophe Mourrat for bringing reference [28] to our attention and an anonymous referee for poiting out to us the preprint [14]. P.C.…”