On the Vocabulary of Grammar-Based Codes and the Logical Consistency of Texts

Abstract

The article presents a new interpretation for Zipf-Mandelbrot's law in natural language which rests on two areas of information theory. Firstly, we construct a new class of grammar-based codes and, secondly, we investigate properties of strongly nonergodic stationary processes. The motivation for the joint discussion is to prove a proposition with a simple informal statement: If a text of length n describes n β independent facts in a repetitive way then the text contains at least n β / log n different words, under suitable conditions on n. In the formal statement, two modeling postulates are adopted. Firstly, the words are understood as nonterminal symbols of the shortest grammar-based encoding of the text. Secondly, the text is assumed to be emitted by a finite-energy strongly nonergodic source whereas the facts are binary IID variables predictable in a shiftinvariant way.

The Relaxed Hilberg Conjecture: A Review and New Experimental Support

“…A few words of clarification are needed here. Roughly speaking, admissibly minimal grammar-based codes are compression algorithms that represent a text as the smallest context-free grammar that generates the text as its sole production (Kieffer and Yang, 2000;Dębowski, 2011b). By the results of Charikar et al (2005), we may suppose that these algorithms are computationally intractable.…”

“…Thus we may expect that Herdan's law for nonterminal symbols is a certain approximation of Herdan's law for words. Investigating mathematical properties of admissibly minimal grammar-based codes, Dębowski (2006Dębowski ( , 2011b showed that Herdan's law for nonterminal symbols is a consequence of the relaxed Hilberg conjecture. Namely, if an arbitrary stationary stochastic process satisfies the relaxed Hilberg conjecture then texts generated by this process satisfy Herdan's law for nonterminal symbols.…”

“…Namely, if an arbitrary stationary stochastic process satisfies the relaxed Hilberg conjecture then texts generated by this process satisfy Herdan's law for nonterminal symbols. Precisely, a heuristic proof of this theorem was given by Dębowski (2006), whereas a mathematically sound demonstration was provided by Dębowski (2011b). Subsequently, Dębowski was interested for which stochastic processes the relaxed Hilberg conjecture is satisfied.…”

On bounded redundancy of universal codes

“…In contrast, by Theorem 7 from Dębowski (2011), the analogous decomposition for the block entropy reads…”

“…Li and Vitányi (2008). While the shortest program for generating a string cannot be efficiently found, there exist also computable universal codes such as the Lempel-Ziv code (Ziv and Lempel, 1977) and grammar-based codes (Kieffer and Yang, 2000;Dębowski, 2011). In particular, the excess length E C µ (n) of admissibly minimal grammar-based codes is bounded above by the number of distinct nonterminal symbols in the grammar used for compression (Dębowski, 2011).…”