A Universal Random Coding Ensemble for Sample-Wise Lossy Compression

Abstract

We propose a universal ensemble for the random selection of rate–distortion codes which is asymptotically optimal in a sample-wise sense. According to this ensemble, each reproduction vector, x^, is selected independently at random under the probability distribution that is proportional to 2−LZ(x^), where LZ(x^) is the code length of x^ pertaining to the 1978 version of the Lempel–Ziv (LZ) algorithm. We show that, with high probability, the resulting codebook gives rise to an asymptotically optimal variable-rate lossy compression scheme under an arbitrary distortion measure, in the sense that a matching converse theorem also holds. According to the converse theorem, even if the decoder knew the ℓ-th order type of source vector in advance (ℓ being a large but fixed positive integer), the performance of the above-mentioned code could not have been improved essentially for the vast majority of codewords pertaining to source vectors in the same type. Finally, we present a discussion of our results, which includes among other things, a clear indication that our coding scheme outperforms the one that selects the reproduction vector with the shortest LZ code length among all vectors that are within the allowed distortion from the source vector.

Lossy Compression of Individual Sequences Revisited: Fundamental Limits of Finite-State Encoders

“…A similar observation applies to [34], where asymptotically pointwise lossy compression was established concerning first-order statistics (i.e., "memoryless" statistics), emphasizing distortion-universality, akin to the focus in [23,24]. A similar fusion of the individual-sequence setting and the probabilistic framework is evident in [35] concerning universal rate-distortion coding. However, akin to the approach in [34], there is no constraint on finite-state encoders/decoders as in [33].…”

“…The main term of the second bound is essentially tighter than the main term of the first bound since Ĥ( Xℓ i ) can be lower bounded by c( xik+k ik+1 ) log c( xik+k ik+1 ), minus some small terms (see, e.g., [35] Equation ( 26)). On the other hand, the second bound is somewhat more complicated due to the introduction of the additional parameter ℓ.…”

“…However, akin to the approach in [34], there is no constraint on finite-state encoders/decoders as in [33]. Notably, the converse theorem in [35] states that for any variable-rate code and any distortion function within a broad class, the vast majority of reproduction vectors representing source sequences of a given type (of any fixed order) must exhibit a code length essentially no smaller than the negative logarithm of the probability of a ball with a normalized radius D (where D denotes the allowed per-letter distortion). This ball is centered at the specified source sequence, and the probability is computed with respect to a universal distribution proportional to 2 −LZ( x) , where LZ( x) denotes the code length of the LZ encoding of the reproduction vector x.…”

“…We compare the performance of the achievability scheme to the ensemble performance of a universal coding scheme that can be implemented when q is exponential in k. The improvement can sometimes be considerably large. The universality of the coding scheme is two-fold: both in the source sequence to be compressed and in the distortion measure in the sense that the order of codewords within the typical codebook (which affects the encoding of their indices) is asymptotically optimal no matter which distortion measure is used (see [35] for a discussion of this property). The intuition behind this improvement is that when q is exponential in k, the memory of the lossless encoder is large enough to store the entire input blocks and thereby exploit the sparseness of the reproduction codebook in the space of k-dimensional vectors with components in the reproduction alphabet.…”

“…The intuition behind this improvement is that when q is exponential in k, the memory of the lossless encoder is large enough to store the entire input blocks and thereby exploit the sparseness of the reproduction codebook in the space of k-dimensional vectors with components in the reproduction alphabet. The asymptotic achievability of the lower bound will rely on the direct coding theorem of [35].…”