Abstract-We consider the estimation of a signal from the knowledge of its noisy linear random Gaussian projections, a problem relevant in compressed sensing, sparse superposition codes or code division multiple access just to cite few. There has been a number of works considering the mutual information for this problem using the heuristic replica method from statistical physics. Here we put these considerations on a firm rigorous basis. First, we show, using a Guerra-type interpolation, that the replica formula yields an upper bound to the exact mutual information. Secondly, for many relevant practical cases, we present a converse lower bound via a method that uses spatial coupling, state evolution analysis and the I-MMSE theorem. This yields, in particular, a single letter formula for the mutual information and the minimal-mean-square error for random Gaussian linear estimation of all discrete bounded signals.Random linear projections and random matrices are ubiquitous in computer science, playing an important role in machine learning [1], statistics [2] and communication [3]. In particular, the task of estimating a signal from its linear random projections has a myriad of applications such as compressed sensing (CS) [4], code division multiple access (CDMA) in communication [5], error correction via sparse superposition codes [6], or Boolean group testing [7]. It is thus natural to ask what are the information theoretic limits for the estimation of a signal via the knowledge of few of its (noisy) random linear projections.A particularly influential approach to this question has been through the use of the heuristic replica method of statistical physics [8], which allows to compute non rigorously the mutual information (MI) and the associated theoretically achievable minimal-mean-square error (MMSE). The replica method typically predicts the optimal performance through the solution of non-linear equations, which interestingly coincide in many cases with the predictions for the performance of a message-passing belief-propagation type algorithm. In this context the algorithm is usually called approximate message-passing (AMP) [9][10][11].In this contribution we prove rigorously that the replica formula for the MI is asymptotically exact for discrete bounded prior distributions of the signal, in the case of random Gaussian linear projections. In particular, our results put on a firm rigorous basis the Tanaka formula for CDMA [12], and allow to rigorously obtain the Bayesian "measurement" MMSE in CS. In addition, our analysis strongly suggests that AMP is reaching the MMSE for a large class of such problems in polynomial time, except for a region called the hard phase. In the hard phase the MMSE can be reached only through the use of a technique called spatial coupling [10,11,13] (SC), originally developed in the context of communication as a practical code construction that allows to reach the Shannon capacity [14]. Finally, we stress that our proof technique has an interest of its own as it is probably transposab...

We recently proved threshold saturation for spatially coupled sparse superposition codes on the additive white Gaussian noise channel [1]. Here we generalize our analysis to a much broader setting. We show for any memoryless channel that spatial coupling allows generalized approximate message-passing (GAMP) decoding to reach the potential (or Bayes optimal) threshold of the code ensemble. Moreover in the large input alphabet size limit: i) the GAMP algorithmic threshold of the underlying (or uncoupled) code ensemble is simply expressed as a Fisher information; ii) the potential threshold tends to Shannon's capacity. Although we focus on coding for sake of coherence with our previous results, the framework and methods are very general and hold for a wide class of generalized estimation problems with random linear mixing.

No abstract

Abstract-Recently, a new class of codes, called sparse superposition or sparse regression codes, has been proposed for communication over the AWGN channel. It has been proven that they achieve capacity using power allocation and various forms of iterative decoding. Empirical evidence has also strongly suggested that the codes achieve capacity when spatial coupling and approximate message passing decoding are used, without need of power allocation. In this note we prove that state evolution (which tracks message passing) indeed saturates the potential threshold of the underlying code ensemble, which approaches in a proper limit the optimal threshold. Our proof uses ideas developed in the theory of low-density parity-check codes and compressive sensing.