Statistical mechanics and capacity-approaching error-correcting codes

Abstract

I will show that there is a deep relation between error-correction codes and certain mathematical models of spin glasses. In particular minimum error probability decoding is equivalent to finding the ground state of the corresponding spin system. The most probable value of a symbol is related to the magnetization at a different temperature. Convolutional codes correspond to one-dimensional spin systems and Viterbi's decoding algorithm to the transfer matrix algorithm of Statistical Mechanics.I will also show how the recently discovered (or rediscovered) capacity approaching codes (turbo codes and low density parity check codes) can be analysed using statistical mechanics. It is possible to show, using statistical mechanics, that these codes allow error-free communication for signal to noise ratio above a certain threshold. This threshold depends on the particular code, and can be computed analytically in many cases. LPTENS 01/ 31It has been known[1 − 4] that error-correcting codes are mathematically equivalent to some theoretical spin-glass models. As it is explained in Forney's paper in this volume, there have been recently very interesting new developments in error-correcting codes. It is now possible to approach practically very close to Shannon's channel capacity. First came the discovery of turbo codes by Berrou and Glavieux [5] and later the rediscovery of low density parity check codes [6], first discovered by Gallager [7,8], in his thesis, in 1962. Both turbo codes and low density parity check (LPDC) codes are based on random structures. It turns out, as I will explain later, that it is possible to use their equivalence with spin glasses, to analyse them using the methods of statistical mechanics.Let me start by fixing the notations. Each information message consists of a sequence of K bits u = {u 1 , · · · , u K }, u i = 0 or 1. The binary vector u is called the source-word. Encoding introduces redundancy into the message. One maps u → x by encoding. u → x has to be a one to one map for the code to be meaningful. The binary vector x has N > K components. It is called a code-word. The ratio R = K/N which specifies the redundancy of the code, is called the rate of the code. One particularly important family of codes are the so-called linear codes. Linear codes are defined by x = G u G is a binary (i.e; its elements are zero or one) (N × K) matrix and the multiplication is modulo two. G is called the generating matrix of the code. Obviously by construction all the components x i of a code-word x are not independent. Of all the 2 N binary vectors only 2 K = 2 N R , those corresponding to a vector u, are code-words. Codewords satisfy the linear constraints (called parity check constraints) H x = 0 (modulo two), where H is a (K ×N ) binary matrix, called the parity check

A Large Deviation Principle for the free energy of random Gibbs measures with application to the REM

“…and therefore lim j→∞ inf σ ∈K c j x≥L J (σ, x) = +∞ for every L. This is condition (11). As to (12), for bounded continuous functions g we have…”

“…Then the condition (10) is verified. Therefore, if (11) is also verified, the tail condition (6) holds true.…”

Applications of Large Random Matrices in Communications Engineering

“…In analogy to classical (commutative) probability theory, all cumulants of the limit distribution P C∞ (x) except for mean and variance vanish. Therefore, the classical central limit theorem can easily be proven along the same lines as (141) and (144). The limit distribution is called the normal distribution, in general, and Gaussian and semicircle distribution for commutative and non-commutative random variables, respectively.…”

“…For further results, the reader is referred to, e.g. [144,145]. The replica method is not the only tool from statistical physics that is applied for the analysis of error correction codes.…”

A schematic model for molecular affinity and binding with Ising variables

“…The connection between error-correcting transmission codes and spin systems was recognized by N. Sourlas [25,26,27]. In particular, it was shown that the usual binary parity checks b 1 ⊕ b 2 ⊕ .…”