A Cognitive Bias for Zipfian Distributions? Uniform Distributions Become More Skewed via Cultural Transmission

Abstract

There is growing evidence that cognitive biases play a role in shaping language structure. Here, we ask whether such biases could contribute to the propensity of Zipfian word-frequency distributions in language, one of the striking commonalities between languages. Recent theoretical accounts and experimental findings suggest that such distributions provide a facilitative environment for word learning and segmentation. However, it remains unclear whether the advantage found in the laboratory reflects prior linguistic experience with such distributions or a cognitive preference for them. To explore this, we used an iterated learning paradigm—which can be used to reveal weak individual biases that are amplified overtime—to see if learners change a uniform input distribution to make it more skewed via cultural transmission. In the first study, we show that speakers are biased to produce skewed word distributions in telling a novel story. In the second study, we ask if this bias leads to a shift from uniform distributions towards more skewed ones using an iterated learning design. We exposed the first learner to a story where six nonce words appeared equally often, and asked them to re-tell it. Their output served as input for the next learner, and so on for a chain of ten learners (or ‘generations’). Over time, word distributions became more skewed (as measured by lower levels of word entropy). The third study asked if the shift will be less pronounced when lexical access was made easier (by reminding participants of the novel word forms), but this did not have a significant effect on entropy reduction. These findings are consistent with a cognitive bias for skewed distributions that gets amplified over time and support the role of entropy minimization in the emergence of Zipfian distributions.

Zipfian Distributions in Child-Directed Speech

“…This skewness might have implications for learning, as suggested by recent studies showing improved learning in such distributions (e.g., Hendrickson & Perfors, 2019 ; Kurumada et al, 2013 ; Lavi-Rotbain & Arnon, 2020 , 2021 , 2022 ). This learnability advantage is also supported by findings showing that speakers turn uniform distributions into skewed ones in the process of cultural transmission, a pattern consistent with the presence of a cognitive bias for them (Shufaniya & Arnon, 2022 ). It has been shown that word segmentation is facilitated in language-like distributions.…”

Injecting structural hints: Using language models to study inductive biases in language learning

“…In Experiment 3, we test the effect of inductive biases in vocabulary distribution: does a bias towards a human-like Zipfian vocabulary distribution (Zipf, 1936) help language learning? Lan-guage is structured not only in how tokens relate, but also in the structure of the distribution that tokens are drawn from, a cognitive bias that is especially significant for memory-based theories of grammar (Shufaniya and Arnon, 2022;Piantadosi, 2014;Ellis and O'Donnell, 2012). We show that a Zipfian pretraining bias makes models better at downstream language learning even when there is no correspondence between the pretraining and fine-tuning vocabularies.…”

“…We next investigate structural biases in the lexicon: how a cognitive bias towards vocabulary distributions (i.e., which tokens are more likely to appear in a corpus of language) affects learning. We aim to answer: does a bias towards a Zipfian vocabulary distribution (Shufaniya and Arnon, 2022;Piantadosi, 2014) act as a structural bias that aids language learning? A feature pervasive across human languages is the unbalanced nature of vocabulary distributions.…”

Cultural evolution creates the statistical structure of language

“…Skewed distributions have also been found to facilitate visual statistical learning (discovering recurring visual triplets in a continuous stream, 20 ); cross-situational word learning 21 ; and learning novel grammatical categories 22 and constructions 23 . Speakers seem to have a cognitive preference for such distributions: uniform distributions of made-up nouns become skewed when a story is transmitted from one participant to another 24 . Though there are relatively few studies that explore the impact of skew on learning, the ones that do, find that skew can be beneficial for learning a range of linguistic and non-linguistic relations.…”