Sparse superposition codes for Gaussian vector quantization

2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

2012

Abstract-We study a new class of codes for Gaussian multiterminal source and channel coding. These codes are designed using the statistical framework of high-dimensional linear regression and are called Sparse Superposition or Sparse Regression codes. Codewords are linear combinations of subsets of columns of a design matrix. These codes were introduced by Barron and Joseph and shown to achieve the channel capacity of AWGN channels with computationally feasible decoding. They have also recently been shown to achieve the optimal rate-distortion function for Gaussian sources. In this paper, we demonstrate how to implement random binning and superposition coding using sparse regression codes. In particular, with minimum-distance encoding/decoding it is shown that sparse regression codes attain the optimal information-theoretic limits for a variety of multiterminal source and channel coding problems.

“…Hence for sufficiently large n, P (E 2 | E c 1 , E c 2 ) < /3. Combining this with (17) and (18), we have P (E) < .…”

Section: Let the Number Of Columns In Each Submentioning

confidence: 71%

Section: Introductionmentioning

confidence: 81%

Sparse regression codes for multi-terminal source and channel coding

2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

2012

“…For every positive integer n, let M n = L b n where L n is determined by (3). Then there exists a sequence C = {C n } n=1,2,... of rate R sparse regression codes -with code C n defined by an n × M n L n design matrix -that attains the optimal error exponent for distortion-level D given by (5).…”

Section: Resultsmentioning

confidence: 99%

“…Here, the performance of these codes under minimum-distance encoding is studied. The design of computationally feasible encoders will be discussed in future work.Sparse regression codes for compressing Gaussian sources were first considered in [3] where some preliminary results were presented. In this paper, we analyze the performance of these codes under optimal (minimum-distance) encoding and show that they can achieve the distortion-rate bound with the optimal error exponent for all rates above a specified value (approximately 1.15 bits/sample).…”

mentioning

confidence: 99%

Gaussian rate-distortion via sparse linear regression over compact dictionaries

Joseph

2012 IEEE International Symposium on Information Theory Proceedings

2012

We study a class of codes for compressing memoryless Gaussian sources, designed using the statistical framework of high-dimensional linear regression. Codewords are linear combinations of subsets of columns of a design matrix. With minimumdistance encoding we show that such a codebook can attain the rate-distortion function with the optimal error-exponent, for all distortions below a specified value. The structure of the codebook is motivated by an analogous construction proposed recently by Barron and Joseph for communication over an AWGN channel. I. INTRODUCTIONOne of the important outstanding problems in information theory is the development of practical codes for lossy compression of general sources at rates approaching Shannon's rate-distortion bound. In this paper, we study the compression of memoryless Gaussian sources using a class of codes constructed based on the statistical framework of high-dimensional linear regression. The codebook consists of codewords that are sparse linear combinations of columns of an n × N design matrix or 'dictionary', where n is the blocklength and N is a low-order polynomial in n. Dubbed Sparse Superposition Codes or Sparse Regression Codes (SPARC), these codes are motivated by an analogous construction proposed recently by Barron and Joseph for communication over an AWGN channel [1], [2]. The structure of the codebook enables the design of computationally efficient encoders based on the rich theory on sparse linear regression and sparse approximation. Here, the performance of these codes under minimum-distance encoding is studied. The design of computationally feasible encoders will be discussed in future work.Sparse regression codes for compressing Gaussian sources were first considered in [3] where some preliminary results were presented. In this paper, we analyze the performance of these codes under optimal (minimum-distance) encoding and show that they can achieve the distortion-rate bound with the optimal error exponent for all rates above a specified value (approximately 1.15 bits/sample). The proof uses Suen's inequality [4], a bound on the tail probability of a sum of dependent indicator random variables. This technique may be of independent interest and useful in other problems in information theory. We lay down some notation before proceeding further. Upper-case letters are used to denote random variables, lowercase for their realizations, and bold-face letters to denote

“…In [8], it was shown that SPARCs achieve the AWGN capacity with feasible decoding. SPARCs for lossy compression were first considered in [9]. In [10], [11], we showed that SPARCs also attain the optimal rate-distortion function of Gaussian sources with feasible algorithms.…”

Section: Introductionmentioning

confidence: 99%

Sparse Regression codes: Recent results and future directions

2013 IEEE Information Theory Workshop (ITW)

2013

Sparse Superposition or Sparse Regression codes were recently introduced by Barron and Joseph for communication over the AWGN channel. The code is defined in terms of a design matrix; codewords are linear combinations of subsets of columns of the matrix. These codes achieve the AWGN channel capacity with computationally feasible decoding. We have shown that they also achieve the optimal rate-distortion function for Gaussian sources. Further, the sparse regression codebook has a partitioned structure that facilitates random binning and superposition. In this paper, we review existing results concerning Sparse Regression codes and discuss directions for future research.