Universal Compression of Memoryless Sources Over Unknown Alphabets

Abstract

For a collection of distributions over a countable support set, the worst case universal compression formulation by Shtarkov attempts to assign a universal distribution over the support set. The formulation aims to ensure that the universal distribution does not underestimate the probability of any element in the support set relative to distributions in the collection. When the alphabet is uncountable and we have a collection P of Lebesgue continuous measures instead, we ask if there is a corresponding universal probability density function (pdf) that does not underestimate the value of the density function at any point in the support relative to pdfs in P. An example of such a measure class is the set of all Gaussian distributions whose mean and variance are in a specified range. We quantify the formulation in the uncountable support case with the attenuation of the class-a quantity analogous to the worst case redundancy of a collection of distributions over a countable alphabet. An attenuation of A implies that the worst case optimal universal pdf at any point x in the support is always at least the value any pdf in the collection P assigns to x divided by A. We analyze the attenuation of the worst optimal universal pdf over length-n samples generated i.i.d. from a Gaussian distribution whose mean can be anywhere between −α/2 to α/2 and variance between σ 2 m and σ 2 M . We show that this attenuation is finite, grows with the number of samples as O(n), and also specify the attentuation exactly without approximations. When only one parameter is allowed to vary, we show that the attenuation grows as O( √ n), again keeping in line with results from prior literature that fix the order of magnitude as a factor of √ n per parameter. In addition, we also specify the attenuation exactly without approximation when only the mean or only the variance is allowed to vary.Keywords: infinitely divisible distributions, universal compression, uncountable support, Gaussians distributions.Compression has been well studied since Shannon [1] formalized not just the notion of what it means to represent data or signals in a compact form, but also quantified how compact the representation can be. For data that come from a countable (discrete) alphabet, this lower bound on compression is essentially the entropy of the source. Furthermore, concrete schemes to represent discrete data in bits are also known-the Huffman coding scheme being the optimal one.While the quantification of the limits of compression is elegant, it does not take into account one of the practicalities of compression-we do not know the underlying distribution. Instead,

Strong Consistency of the Good-Turing Estimator

“…This establishes (1). Since nP n (X n ) converges in distribution to Y , we can create a sequence of random variables {Y n } ∞ n=1…”

Probability Estimation in the Rare-Events Regime

“…He noted that his result implies that the class of stationary and ergodic sources over an infinite alphabet is not universally compressible, in contrast to the finite-alphabet case. The impetus for the present paper came from the more-recent work of Orlitsky et al [11], [12]. These authors called attention to the discrepancy between the asymptotic that is typically used in information theory and many realistic data sources, and showed that a function of the observed sequence called the pattern can be universally compressed, even when the alphabet is infinite and the source distribution is arbitrary.…”

A Lower-Bound for the Maximin Redundancy in Pattern Coding

“…It was first introduced byÅberg in [8] as a solution to the multi-alphabet coding problem, where the message x contains only a small subset of the known alphabet A. It was further studied and motivated in a series of articles by Shamir [9][10][11][12] and by Jevtić, Orlitsky, Santhanam and Zhang [13][14][15][16] for practical applications: the alphabet is unknown and has to be transmitted separately anyway (for instance, transmission of a text in an unknown language), or the alphabet is very large in comparison to the message (consider the case of images with k = 2 24 colors, or texts when taking words as the alphabet units).…”

“…Using the same notations as in [15], we call multiplicity µ j (ψ) of symbol j in pattern ψ ∈ P n the number of occurrences of j in ψ; the multiplicity of pattern ψ is the vector made of all symbol's multiplicities:…”

“…As explained in [15], there is one-to-one mapping between Φ n and the set of unordered partitions of integer n. In Section 3., this point will be used and specified.…”