783A&ruct-Universal coding is any asymptotically optimum method of block-to-block memoryless source coding for sources with unknown parameters. This paper considers noiseless coding for such sources, primarily in terms of variable-length coding, with performance measured as a function of the coding redundancy relative to the per-letter conditional source entropy given the unknown parameter. It is found that universal (i.e., zero redundancy) coding in a weighted sense is possible if and only if the per-letter average mutual information between the parameter space and the message space is zero. Universal coding is possible in a maximin sense if and only if the channel capacity between the two spaces is zero. Universal coding is possible in a minimax sense if and only if a probability mass function exists, independent of the unknown parameter, for which the relative entropy of the known conditional-probability mass-function is zero. Several examples are given to illustrate the ideas. Particular attention is given to sources that are stationary and ergodic for any fixed parameter although the whole ensemble is not. For such sources, weighted universal codes always exist if the alphabet is finite, or more generally if the entropy is finite. Minimax universal codes result if an additional entropy stability constraint is applied. A discussion of fixed-rate universal coding is also given briefly with performance measured by a probability of error.

The most recent (summer 1999) revision fixed numerous typos reported during the previous year and added quite a bit of material on jointly Gaussian vectors in Chapters 3 and 4 and on minimum mean squared error estimation of vectors in Chapter 4.This revision is a work in progress. Revised versions will be made available through the World Wide Web page http://www-isl.stanford.edu/~gray/sp.html . The material is copyrighted by the authors, but is freely available to any who wish to use it provided only that the contents of the entire text remain

No abstract

No abstract

No abstract