An information-theoretic approach to assess linguistic complexity

The Entropy of Words—Learnability and Expressivity across More than 1000 Languages

“…Displacing the distribution to the left or skewing it towards low values would compromise expressivity. In contrast, skewing it towards the right increases the potential for expressivity according to Equation (13), though this comes with a learnability cost.…”

“…For example, several recent studies engage in establishing information-theoretic and corpus-based methods for linguistic typology, i.e., classifying and comparing languages according to their information encoding potential [10,[12][13][14][15][16], and how this potential evolves over time [17][18][19]. Similar methods have been applied to compare and distinguish non-linguistic sequences from written language [20,21], though it is controversial whether this helps with more fine-grained distinctions between symbolic systems and written language [22,23].…”

“…In the context of quantitative linguistics, entropic measures are used to understand laws in natural languages, such as the relationship between word frequency, predictability and the length of words [24][25][26][27], or the trade-off between word structure and sentence structure [10,13,28]. Information theory can further help to understand the complexities involved when building words from the smallest meaningful units, i.e., morphemes [29,30].…”

The Entropy of Words—Learnability and Expressivity across More Than 1000 Languages

“…For example, several recent studies engage in establishing information-theoretic and corpus-based methods for linguistic typology, i.e. classifying and comparing languages according to their information encoding potential [10,[12][13][14][15][16], and how this potential evolves over time [17][18][19]. Similar methods have been applied to compare and distinguish non-linguistic sequences from written 2 of 34 language [20,21], though it is controversial whether this helps with more fine-grained distinctions between symbolic systems and written language [22,23].…”

“…In the context of quantitative linguistics, entropic measures are used to understand laws in natural languages, such as the relationship between word frequency, predictability and the length of words [24][25][26][27], or the trade-off between word structure and sentence structure [10,13,28]. Information-theory can further help to understand the complexities involved when building words from the smallest meaningful units, i.e.…”