On the universality of noiseless linear estimation with respect to the measurement matrix

Abstract

In a noiseless linear estimation problem, one aims to reconstruct a vector x * from the knowledge of its linear projections y = Φx * . There have been many theoretical works concentrating on the case where the matrix Φ is a random i.i.d. one, but a number of heuristic evidence suggests that many of these results are universal and extend well beyond this restricted case. Here we revisit this problematic through the prism of development of message passing methods, and consider not only the universality of the 1 transition, as previously addressed, but also the one of the optimal Bayesian reconstruction. We observed that the universality extends to the Bayes-optimal minimum mean-squared (MMSE) error, and to a range of structured matrices.

Universality of Approximate Message Passing with Semi-Random Matrices

“…Since their inception, AMP algorithms have found applications in diverse situations-on the one hand, they are directly used as computationally efficient inference algorithms in compressed sensing [25] and coding theory [62]; on the other hand, these algorithms have been used as constructive proof devices to characterize the asymptotic performance of statistical procedures such as the LASSO [5], M-estimators [7,22,35,34], maximum likelihood [67,66], and spectral methods [54,52] in high-dimensions. Given a data matrix M ∈ R N ×N , an AMP algorithm in its general form consists of the following iterative updates: z (t) = M F t (z (0) , z (1) , . .…”

“…AMP algorithms are particularly attractive due to their theoretical tractability. Specifically, when the data matrix M is drawn from a rotationally-invariant ensemble (such as the Gaussian orthogonal ensemble), and if the function G t (called the Onsager correction) is suitably chosen based on F t , the joint empirical distributions of the iterates z (1) , . .…”

“…Interestingly, numerical experiments suggest that the behavior of AMP algorithms does not depend too strongly on the precise distribution of the matrix M . In fact, it has been observed that [16,1,44] the theoretical characterizations obtained under idealistic statistical models remain true for many semi-random (or even deterministic) matrix ensembles. Establishing this universality phenomenon is thus of intrinsic importance, as it allows practitioners to use these theoretical characterizations of AMP with greater confidence in real-life statistical and machine learning applications.…”

“…In the context of AMP algorithms, numerical simulations also suggest the equivalence between the Sine model and the ROM, in that they can be characterized by the same state evolution recursion (see Section 2.3 for supporting numerical evidence). More generally, numerical studies reported in the literature [1,16] suggest that AMP algorithms exhibit universality properties as long as the eigenvectors of M are generic. Formalizing this conjecture remains squarely beyond existing techniques, and presents a fascinating challenge.…”

“…We wish to understand the dynamics of the above algorithm under general assumptions on the matrix M , the (coordinate-wise) non-linearities f t and the initialization z (0) . The algorithm in (2) is a special case of (1). The specific choice of F t made in (1) to obtain (2) ensures that the iterates of the resulting algorithm converge to a Gaussian process as N → ∞ without any Onsager correction (given by the function G t in (1)).…”