Generalized Context Transformations -- Enhanced Entropy Reduction

Abstract

Generalized Context Transformations (GCT) is a transformation method capable to reduce zero-order entropy H by removing a mutual information I between digrams and individual symbols distribution. GCT is an improved version of the Context Transformation(CT) method presented recently [1]. GCT is used as a preprocessor for zero-order entropy coding algorithms like Arithmetic or Huffman coding. The transformation itself is based on digrams exchange. In the former CT algorithm, digrams αβ and αγ were exchanged, when p(α, β) = 0, p(α, γ) = 0 and p(γ) > p(β). This approach has led to the entropy reduction, because each symbol β that follows α is coded like it would be a more probable symbol γ, but in the algorithm design there were some inefficiencies that we removed in the new design of GCT algorithm. The main improvement in GCT is that we are now able to predict the entropy change before the particular transformation is applied on the source. GCT is no more restricted on cases, when p(α, γ) = 0, we are now able to exhange any digram for any digram, it doesn't matter if they are present in the source or not, but due to the transformation search complexity we decided to use only digrams that share the context symbol α. Algorithms were examined on the Calgary Corpus and in comparison with the former CT algorithm we were able to achieve significantly better entropy reduction and we were also able to reduce entropy in cases when our former CT algorithm was unable to reduce entropy at all.

Prediction and Evaluation of Zero Order Entropy Changes in Grammar-Based Codes

“…Our recent studies were focused on a special class of grammar transforms that leave the message size intact [19,20]. In the present paper, the class of grammar transformations is extended with a novel concept of higher order context transformation [21].…”

“…This restriction is removed by the introduction of the generalized context transformations first proposed in [20]. Definition 2.…”

“…If we apply one of transformations, then ∆m = 0, and as a consequence, c 1 = 1; the last term on the right will be zero due to log c 1 = 0, so Equation (20) tells us that the entropic size of the message carried by these symbols does not change during transformation. When |Σ T | is much smaller than |Σ I |, it is convenient to rewrite Equation (20) in terms of Σ T :…”

“…When |Σ T | is much smaller than |Σ I |, it is convenient to rewrite Equation (20) in terms of Σ T :…”