Universal noiseless coding

Abstract

783A&ruct-Universal coding is any asymptotically optimum method of block-to-block memoryless source coding for sources with unknown parameters. This paper considers noiseless coding for such sources, primarily in terms of variable-length coding, with performance measured as a function of the coding redundancy relative to the per-letter conditional source entropy given the unknown parameter. It is found that universal (i.e., zero redundancy) coding in a weighted sense is possible if and only if the per-letter average mutual information between the parameter space and the message space is zero. Universal coding is possible in a maximin sense if and only if the channel capacity between the two spaces is zero. Universal coding is possible in a minimax sense if and only if a probability mass function exists, independent of the unknown parameter, for which the relative entropy of the known conditional-probability mass-function is zero. Several examples are given to illustrate the ideas. Particular attention is given to sources that are stationary and ergodic for any fixed parameter although the whole ensemble is not. For such sources, weighted universal codes always exist if the alphabet is finite, or more generally if the entropy is finite. Minimax universal codes result if an additional entropy stability constraint is applied. A discussion of fixed-rate universal coding is also given briefly with performance measured by a probability of error.

Adaptive context trees and text clustering

“…The counters and are defined as before by (3). Therefore, the distribution can also be defined as before by (4).…”

Learning and storing the parts of objects: IMF

“…The statistics of the data ensemble may be used to compress the data ensemble; however, prior knowledge of the statistics of the factorization may not be assumed as NMF is non-unique (without performing multiple passes through the factorization, which is not practical in a alternating minimization routine). Universal coding schemes address this problem by coupling learning with the coding process for varying source characteristics [14,15]. We take the 1977 approach of Lempel and Ziv [2] (LZ77) for postfactorization-coding, even though encoding ratios may be slightly improved using more recent extensions.…”

The cost of uncorrelation and non-cooperation in MIMO channels

“…A more optimistic approach, very common in the areas of universal source coding [11] and universal channel decoding [12], [13], is that of competitive optimality. Instead of trying to guarantee a certain rate, we look for a robust input P * * which for any channel W ∈ W does not loose "too much" mutual information relative to the channel capacity, i.e.,…”