Approximate Message Passing algorithms for rotationally invariant matrices

Abstract

Approximate Message Passing (AMP) algorithms have seen widespread use across a variety of applications. However, the precise forms for their Onsager corrections and state evolutions depend on properties of the underlying random matrix ensemble, limiting the extent to which AMP algorithms derived for white noise may be applicable to data matrices that arise in practice.In this work, we study more general AMP algorithms for random matrices W that satisfy orthogonal rotational invariance in law, where W may have a spectral distribution that is different from the semicircle and Marcenko-Pastur laws characteristic of white noise. The Onsager corrections and state evolutions in these algorithms are defined by the free cumulants or rectangular free cumulants of the spectral distribution of W. Their forms were derived previously by Opper, Çakmak and Winther using nonrigorous dynamic functional theory techniques, and we provide rigorous proofs.Our motivating application is a Bayes-AMP algorithm for Principal Components Analysis, when there is prior structure for the principal components (PCs) and possibly nonwhite noise. For sufficiently large signal strengths and any non-Gaussian prior distributions for the PCs, we show that this algorithm provably achieves higher estimation accuracy than the sample PCs.

A unifying tutorial on Approximate Message Passing

“…The proofs we give for H k+1 (c, d) combine aspects of the asymptotic and finite-sample arguments (see Remark 6.3) in the existing AMP literature. Proposition E.1 in Fan (2020) provides the basis for an alternative asymptotic approach, whose details we omit.…”

“…Gaussian entries, AMP is not guaranteed to converge, and in fact can even diverge in sometimes pathological ways; see Rangan et al (2019a) for a discussion of this issue. For this reason, a number of other AMP-based algorithms have been introduced that allow for this assumption to be weakened in various ways, such as Vector AMP (VAMP) (Rangan et al, 2019b), orthogonal AMP (OAMP) (Ma and Ping, 2017;Takeuchi, 2020) and other generalisations of AMP for rotationally invariant matrices (Opper et al, 2016;Fan, 2020).…”

“…Mehta, 2004). For other orthogonally invariant W , Fan (2020) showed that the original symmetric AMP algorithm ( 22) can be modified to allow each v k+1 to depend on all of the previous iterates via…”

“…The estimation problem here is equivalent to one of reconstructing a symmetric rank-one matrix in a Gaussian spiked model. Bolthausen (2014) proved a rigorous state evolution result for AMP in this specific setting, by introducing a conditioning argument that became an essential ingredient in subsequent analyses of AMP (Bayati and Montanari, 2011;Javanmard and Montanari, 2013;Berthier et al, 2020;Fan, 2020). See Section 6.2 for a detailed discussion of this proof technique.…”

“…For matrices with independent sub-Gaussian entries, results on the 'universality' of AMP were first established by Bayati et al (2015) and later in greater generality by Chen and Lam (2021). In addition, to accommodate the class of rotationally invariant random matrices, a number of extensions of the original AMP framework have recently been proposed, including Orthogonal AMP (Ma and Ping, 2017;Takeuchi, 2020) and Vector AMP (Schniter et al, 2016;Rangan et al, 2019b), as well as the general iterative schemes of Opper et al (2016), C ¸akmak and Opper (2019) and Fan (2020). Some of these are closely related to expectation propagation (Opper and Winther, 2005;Kabashima and Vehkaperä, 2014).…”