A large quantitative analysis of written language challenges the idea that all languages are equally complex

Abstract

One of the fundamental questions about human language is whether all languages are equally complex. Here, we approach this question from an information-theoretic perspective. We present a large scale quantitative cross-linguistic analysis of written language by training a language model on more than 6500 different documents as represented in 41 multilingual text collections consisting of ~ 3.5 billion words or ~ 9.0 billion characters and covering 2069 different languages that are spoken as a native language by more than 90% of the world population. We statistically infer the entropy of each language model as an index of what we call average prediction complexity. We compare complexity rankings across corpora and show that a language that tends to be more complex than another language in one corpus also tends to be more complex in another corpus. In addition, we show that speaker population size predicts entropy. We argue that both results constitute evidence against the equi-complexity hypothesis from an information-theoretic perspective.

Languages with more speakers tend to be harder to (machine-)learn

“…Yet, depending on how you count, there are between 6000 and 8000 different languages and language varieties on the planet 13 – 15 that vary widely in their structural properties 16 , 17 . A growing body of cross-linguistic research has begun to document that the natural and social environments in which languages are being used and learned drive this diversity 18 – 21 , that language structure is influenced by socio-demographic factors such as the estimated number of speakers 18 , 21 – 23 and that the long-held belief in a principle of "invariance of language complexity" 24 may be incorrect 25 .…”

“…faster) to learn: in a large-scale quantitative cross-linguistic analysis, Ref. 25 trained an LM on more than 6500 documents in over 2000 different languages and statistically inferred the entropy rate of each document, which can be seen as an index of the underlying language complexity 28 , 33 – 35 . The results showed that documents in languages with more speakers tended to be more complex.…”

“…In this article, we first use part of the data used by Ref. 25 to explicitly test this hypothesis. Since the LM used by Ref.…”

“…Since the LM used by Ref. 25 is rather simple, we train two more sophisticated LMs that use machine learning and deep learning on the data and compare the results. We then discuss some potential limitations of the multilingual text collection used by Ref.…”

“…We then discuss some potential limitations of the multilingual text collection used by Ref. 25 . To rule out that the results are driven by these limitations, we create two fully balanced and parallel multilingual corpora, which we use to train seven different LMs—ranging from very simple n-gram models to state-of-the-art deep neural networks—and measure how difficult it is for each LM to build an adequate probabilistic representation of the input.…”