Excess entropy in natural language: Present state and perspectives

Abstract

We review recent progress in understanding the meaning of mutual information in natural language. Let us define words in a text as strings that occur sufficiently often. In a few previous papers, we have shown that a power-law distribution for so defined words (a.k.a. Herdan's law) is obeyed if there is a similar power-law growth of (algorithmic) mutual information between adjacent portions of texts of increasing length. Moreover, the power-law growth of information holds if texts describe a complicated infinite (algorithmically) random object in a highly repetitive way, according to an analogous power-law distribution. The described object may be immutable (like a mathematical or physical constant) or may evolve slowly in time (like cultural heritage). Here, we reflect on the respective mathematical results in a less technical way. We also discuss feasibility of deciding to what extent these results apply to the actual human communication.

The Relaxed Hilberg Conjecture: A Review and New Experimental Support

“…However, these later results may be also highly interesting for linguists and thus worth popularizing in this venue. A similar review for physicists and researchers working in complex systems was published by Dębowski (2011a).…”

Infinite Excess Entropy Processes with Countable-State Generators

“….. It has a long history and is widely employed as a measure of correlation and complexity in a variety of fields, from ergodic theory and dynamical systems to neuroscience and linguistics [1][2][3][4][5][6]. For a review the reader is referred to [7].…”

“…Indeed, one can show that any process generated by a stationary, countable-state HMM either has positive entropy rate or consists entirely of periodic sequences, which these do not. Versions of the Santa Fe Process introduced in [6] are finite-alphabet, infinitary processes with positive entropy rate. However, they were not constructed directly as hidden Markov processes, and it seems unlikely that they should have any stationary, countable-state presentations either.…”

Predictive Rate-Distortion for Infinite-Order Markov Processes

“…Many natural processes in physics, biology, neuroscience, finance, and quantitative social science are highly non-Markovian with slowly asymptoting or divergent E [40]. This implies rather small spectral gaps if the process has a countable infinity of causal states-e.g., as in Ref.…”