Optimal Coding and the Origins of Zipfian Laws

Abstract

The problem of compression in standard information theory consists of assigning codes as short as possible to numbers. Here we consider the problem of optimal coding -under an arbitrary coding scheme -and show that it predicts Zipf's law of abbreviation, namely a tendency in natural languages for more frequent words to be shorter. We apply this result to investigate optimal coding also under so-called nonsingular coding, a scheme where unique segmentation is not warranted but codes stand for a distinct number. Optimal non-singular coding predicts that the length of a word should grow approximately as the logarithm of its frequency rank, which is again consistent with Zipf's law of abbreviation. Optimal non-singular coding in combination with the maximum entropy principle also predicts Zipf's rank-frequency distribution. Furthermore, our findings on optimal non-singular coding challenge common beliefs about random typing. It turns out that random typing is in fact an optimal coding process, in stark contrast with the common assumption that it is detached from cost cutting considerations. Finally, we discuss the implications of optimal coding for the construction of a compact theory of Zipfian laws more generally as well as other linguistic laws.

Adherence to Menzerath's Law is the exception (not the rule) in three duetting primate species

“…Despite the prevalence of Menzerath's Law across diverse biological systems [69] and previous reports documenting patterns consistent with the law in chimpanzee [32] and male gibbon [20,21] vocalizations, we found mixed support for Menzerath's Law in primate duets. As adherence to Menzerath's Law is a prediction of compression, it may be that selection pressures related to vocal plasticity or biomechanical constraints on note production have been more important in shaping primate duets than selection for compression or efficiency.…”

The learnability consequences of Zipfian distributions: Word Segmentation is Facilitated in More Predictable Distributions

“…That is, the particular shape of the Zipfian distribution may confer a unique learnability advantage with large enough lexicons, by making words more distinguishable and allowing for stable predictability values for varying samples and sample sizes. These cognitive benefits are joined by communicative pressures: such distributions are claimed to create an optimal trade-off between speaker and listener effort (33). We are currently investigating these possibilities using computational simulations, mathematical modelling, and expanded word segmentation paradigms.…”

Comparing phonological and orthographic networks: A multiplex analysis

“…Many studies focused on the complexity of natural language have pointed out that language is the manifestation of different levels of complex organization [ 1 – 4 ], ranging from semantics [ 5 ] to syntax [ 6 , 7 ] or even emotional components [ 8 ]. Of particular interest are the applications of network science on language organization, where these levels of complexity may be explored by means of single [ 9 , 10 ] and multilayer graphs [ 11 , 12 ].…”