Threshold saturation of spatially coupled sparse superposition codes for all memoryless channels

Barbier, Jean; Dia, Mohamad; Macris, Nicolas

doi:10.1109/itw.2016.7606799

Cited by 24 publications

(57 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Definition II.1 (Asymptotic distortion). Consider the vectors x n×1 andx n×1 defined over n , and let d(·; ·) be a distortion function defined in (24). Define the index set Ï(n) as a subset of [1 : n], and let |Ï(n)| grow with n such that 13 for some η ∈ [0, 1].…”

Section: B Asymptotic Distortion and Conditional Distributionmentioning

confidence: 99%

Statistical Mechanics of MAP Estimation: General Replica Ansatz

Bereyhi

Müller

Schulz-Baldes

2019

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

The large-system performance of maximum-a-posterior estimation is studied considering a general distortion function when the observation vector is received through a linear system with additive white Gaussian noise. The analysis considers the system matrix to be chosen from the large class of rotationally invariant random matrices. We take a statistical mechanical approach by introducing a spin glass corresponding to the estimator, and employing the replica method for the large-system analysis. In contrast to earlier replica based studies, our analysis evaluates the general replica ansatz of the corresponding spin glass and determines the asymptotic distortion of the estimator for any structure of the replica correlation matrix. Consequently, the replica symmetric as well as the replica symmetry breaking ansatz with b steps of breaking is deduced from the given general replica ansatz. The generality of our distortion function lets us derive a more general form of the maximum-a-posterior decoupling principle. Based on the general replica ansatz, we show that for any structure of the replica correlation matrix, the vector-valued system decouples into a bank of equivalent decoupled linear systems followed by maximum-a-posterior estimators. The structure of the decoupled linear system is further studied under both the replica symmetry and the replica symmetry breaking assumptions. For b steps of symmetry breaking, the decoupled system is found to be an additive system with a noise term given as the sum of an independent Gaussian random variable with b correlated impairment terms. The general decoupling property of the maximum-a-posterior estimator leads to the idea of a replica simulator which represents the replica ansatz through the state evolution of a transition system described by its corresponding decoupled system. As an application of our study, we investigate large compressive sensing systems by considering the ℓp norm minimization recovery schemes. Our numerical investigations show that the replica symmetric ansatz for ℓ0 norm recovery fails to give an accurate approximation of the mean square error as the compression rate grows, and therefore, the replica symmetry breaking ansätze are needed in order to assess the performance precisely. Index TermsMaximum-a-posterior estimation, linear vector channel, decoupling principle, equivalent single-user system, compressive sensing, zero norm, replica method, statistical physics, replica symmetry breaking, replica simulatorThe results of this manuscript were presented in parts at 2016 IEEE Information Theory Workshop (ITW) [78] and 2017 IEEE Information Theory and Applications Workshop (ITA) [79]. ). 2 then, expresses the large-system performance regarding the distortion function d(·; ·). The performance analysis of the estimator requires (2) to be explicitly computed, and then,x = g(y|A) substituted in the distortion function.This task, however, is not trivial for many choices of the utility function and the source support , and becomes unfeasible as n grows large....

show abstract

Section: B Asymptotic Distortion and Conditional Distributionmentioning

confidence: 99%

Statistical Mechanics of MAP Estimation: General Replica Ansatz

Bereyhi

Müller

Schulz-Baldes

2019

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…In this paper, we only consider the standard SPARC construction described in Section I-A. Extending the analysis to the spatially-coupled SPARCs proposed in [8], [21], [22] is an interesting research direction and part of ongoing work.…”

Section: Structure Of the Paper And Main Contributionsmentioning

confidence: 99%

The Error Probability of Sparse Superposition Codes With Approximate Message Passing Decoding

Rush

Venkataramanan

2019

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Sparse superposition codes, or sparse regression codes (SPARCs), are a recent class of codes for reliable communication over the AWGN channel at rates approaching the channel capacity. Approximate message passing (AMP) decoding, a computationally efficient technique for decoding SPARCs, has been proven to be asymptotically capacity-achieving for the AWGN channel. In this paper, we refine the asymptotic result by deriving a large deviations bound on the probability of AMP decoding error. This bound gives insight into the error performance of the AMP decoder for large but finite problem sizes, giving an error exponent as well as guidance on how the code parameters should be chosen at finite block lengths. For an appropriate choice of code parameters, we show that for any fixed rate less than the channel capacity, the decoding error probability decays exponentially in n/(log n) 2T , where T , the number of AMP iterations required for successful decoding, is bounded in terms of the gap from capacity. these strong theoretical guarantees, the rates achieved by this decoder for practical block lengths are significantly less than C. Subsequently, a adaptive soft-decision iterative decoder was proposed by Cho and Barron [3], with improved finite length performance for rates closer to capacity. Theoretically, the decoding error probability of the adaptive soft-decision decoder was shown to decay exponentially in n/(log n) 2T , where T is the minimum number of iterations [4], [5].Recently, decoders for SPARCs based on Approximate Message Passing (AMP) techniques were proposed in [6]- [8]. AMP decoding has several attractive features, notably, the absence of tuning parameters, its superior empirical performance at finite block lengths, and its low complexity when implemented using implicitly defined Hadamard design matrices [7], [8]. Furthermore, its decoding performance in each iteration can be predicted using a deterministic scalar iteration called 'state evolution'.In this paper, we provide a non-asymptotic analysis of the AMP decoder proposed in [7]. In [7], it was proved that the state evolution predictions for the AMP decoder are asymptotically accurate, and that for any fixed rate R < C, the probability of decoding error goes to zero with growing block length. However this result did not specify the rate of decay of the probability of error. In this paper, we refine the asymptotic result in [7], and derive a large deviations bound for the probability of error of the AMP decoder (Theorem 1). This bound gives insight into the error performance of the AMP decoder for large but finite problem sizes, giving an error exponent as well as guidance on how the code parameters should be chosen at finite block lengths.The error probability bound for the AMP decoder is of the same order as the bound for the Cho-Barron softdecision decoder [4], [5]: both bounds decay exponentially in n/(log n) 2T , where T is the minimum number of iterations. However, the AMP decoder has slightly lower complexity and has been empirically found to h...

show abstract

“…Computing mutual informations in GLMs is also a critical issue in con rming the information bottleneck scenario of [31,32] d) In communications, error-correcting codes that use random constructions are particularly e cient, as discussed by Shannon in his seminal paper [33]. Random instances of GLMs describe both the setting of codedivision multiple access -a multi-user access method used in communication technologies [34,35]-as well as an error correction scheme called sparse superposition codes, that have been shown to achieve the Shannon capacity for any type of noisy channel [36][37][38][39][40].…”

Section: Main Part 1 Introductionmentioning

confidence: 99%

Optimal errors and phase transitions in high-dimensional generalized linear models

Barbier

Krząkała

Macris

et al. 2019

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

179

318

View full text Add to dashboard Cite

Generalized linear models (GLMs) are used in high-dimensional machine learning, statistics, communications, and signal processing. In this paper we analyze GLMs when the data matrix is random, as relevant in problems such as compressed sensing, error-correcting codes, or benchmark models in neural networks. We evaluate the mutual information (or “free entropy”) from which we deduce the Bayes-optimal estimation and generalization errors. Our analysis applies to the high-dimensional limit where both the number of samples and the dimension are large and their ratio is fixed. Nonrigorous predictions for the optimal errors existed for special cases of GLMs, e.g., for the perceptron, in the field of statistical physics based on the so-called replica method. Our present paper rigorously establishes those decades-old conjectures and brings forward their algorithmic interpretation in terms of performance of the generalized approximate message-passing algorithm. Furthermore, we tightly characterize, for many learning problems, regions of parameters for which this algorithm achieves the optimal performance and locate the associated sharp phase transitions separating learnable and nonlearnable regions. We believe that this random version of GLMs can serve as a challenging benchmark for multipurpose algorithms.

show abstract

Threshold saturation of spatially coupled sparse superposition codes for all memoryless channels

Cited by 24 publications

References 17 publications

Statistical Mechanics of MAP Estimation: General Replica Ansatz

Statistical Mechanics of MAP Estimation: General Replica Ansatz

The Error Probability of Sparse Superposition Codes With Approximate Message Passing Decoding

Optimal errors and phase transitions in high-dimensional generalized linear models

Contact Info

Product

Resources

About