Natural language from function dynamics

Kataoka, Naoto; Kaneko, Kunihiko

doi:10.1016/s0303-2647(00)00079-4

Cited by 6 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, we expect that fundamental structures underlying natural language can be anatomized by analyzing the dynamics of the pre-trained ALBERT model. There have been several works that attempt to analyze the structure of natural language using dynamical systems theory [10][11][12] and chaotic dynamics [13][14][15][16]. Our study aims to advance these dynamical systems approaches to understand natural language, by using the up-to-date machine learning model.…”

Section: A Albert As "The Reservoir"mentioning

confidence: 99%

Transient Chaos in BERT

Inoue,

Ohara,

Kuniyoshi

et al. 2021

Preprint

View full text Add to dashboard Cite

Language is an outcome of our complex and dynamic human-interactions and the technique of natural language processing (NLP) is hence built on human linguistic activities. Along with Generative Pre-trained Transformer (GPT), Bidirectional Encoder Representations from Transformers (BERT) has recently gained its popularity, owing to its outstanding NLP capability, by establishing the state-of-the-art scores in several NLP benchmarks. A Lite BERT (ALBERT) is literally characterized as a lightweight version of BERT, in which the number of BERT parameters is reduced by repeatedly applying the same neural network called Transformer's encoder layer. By pre-training the parameters with a massive amount of natural language data, ALBERT can convert input sentences into versatile high-dimensional vectors potentially capable of solving multiple NLP tasks. In that sense, ALBERT can be regarded as a well-designed high-dimensional dynamical system whose operator is the Transformer's encoder, and essential structures of human language are thus expected to be encapsulated in its dynamics. In this study, we investigated the embedded properties of ALBERT to reveal how NLP tasks are effectively solved by exploiting its dynamics. We thereby aimed to explore the nature of human language from the dynamical expressions of the NLP model. Our analysis consists of two parts, namely short-and longterm analyses, according to time-scale differences to capture the dynamics. Our short-term analysis clarified that the pre-trained model stably yields trajectories with higher dimensionality in a certain time range, which would enhance the expressive capacity required for NLP tasks. Also, our long-term analysis revealed that AL-BERT intrinsically shows transient chaos, a typical nonlinear phenomenon showing chaotic dynamics only in its transient, and the pre-trained ALBERT model tends to produce the chaotic trajectory for a significantly longer time period compared to a randomly-initialized one. Our results imply that local chaoticity would contribute to improving NLP performance, uncovering a novel aspect in the role of chaotic dynamics in human language behaviors.

show abstract

Section: A Albert As "The Reservoir"mentioning

confidence: 99%

Transient Chaos in BERT

Inoue,

Ohara,

Kuniyoshi

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Around the same time, similar methodologies became popular to study the dynamics of language change, i.e. the replacement of already established conventions, both in general (Niyogi andBerwick, 1995, 1997;Arita and Koyama, 1998;Nettle, 1999;Kataoka and Kaneko, 2000;Livingstone, 2000;Ritt, 2004;De Oliveira et al, 2005;Niyogi, 2006;Wedel, 2006;Baxter et al, 2006;Wedel, 2007;Ettlinger, 2007a,b;Fagyal et al, 2010;Blythe and Croft, 2012;Gong et al, 2012;Otero-Espinar et al, 2013;Sóskuthy, 2013;Pierrehumbert et al, 2014;Enke et al, 2016;Kauhanen and Walkden, 2015) as well as for some specific historical changes in particular (Yang, 2002;Choudhury et al, 2006Choudhury et al, , 2007Pearl and Weinberg, 2007;Troutman et al, 2008;Baxter et al, 2009;Sonderegger and Niyogi, 2010;Swarup and McCarthy, 2012;Ritt and Baumann, 2012;Kirby, 2013;Kirby and Sonderegger, 2013).…”

Section: Why Model?mentioning

confidence: 99%

Momentum in Language Change

Stadler

Blythe²,

Smith

et al. 2016

Lang Dyn Change

View full text Add to dashboard Cite

Like other socially transmitted traits, human languages undergo cultural evolution. While humans can replicate linguistic conventions to a high degree of fidelity, sometimes established conventions get replaced by new variants, with the gradual replacement following the trajectory of an s-shaped curve. Although previous modelling work suggests that only a bias favoring the replication of new linguistic variants can reliably reproduce the dynamics observed in language change, the source of this bias is still debated. In this paper we compare previous accounts with a momentum-based selection account of language change, a replicator-neutral model where the popularity of a variant is modulated by its momentum, i.e. its change in frequency of use in the recent past. We present results from a multi-agent model that are characteristic of language change, in particular by exhibiting spontaneously generated s-shaped transitions that do not require externally triggered actuation. We discuss several empirical questions raised by our model, pertaining to both momentum-based selection as well as other biases and pressures which have been suggested to influence language change.

show abstract

“…Following an analogy with language, elementary CA rules act like words which are used in PICARD macroexecutions. Using a functional-dynamics framework as a model for the evolution of natural language, Kataoka and Kaneko (2000b) show how self-referencing function dynamics can act like a filter that produces a corpus of words that are each fixed points of the dynamical model. The invariant macrostate CA rules that we describe are qualitatively similar to this idea, and their representation is consistent with measured distributions of words in actual natural language.…”

Section: A Picard Case Studymentioning

confidence: 99%

Self-Referencing Cellular Automata: A Model of the Evolution of Information Control in Biological Systems

Pavlic¹,

Adams

Davies³

et al. 2014

Artificial Life 14: Proceedings of the Fourteenth International Conference on the Synthesis and Simulation of Living Systems

View full text Add to dashboard Cite

Cellular automata have been useful artificial models for exploring how relatively simple rules combined with spatial memory can give rise to complex emergent patterns. Moreover, studying the dynamics of how rules emerge under artificial selection for function has recently become a powerful tool for understanding how evolution can innovate within its genetic rule space. However, conventional cellular automata lack the kind of state feedback that is surely present in natural evolving systems. Each new generation of a population leaves an indelible mark on its environment and thus affects the selective pressures that shape future generations of that population. To model this phenomenon, we have augmented traditional cellular automata with state-dependent feedback. Rather than generating automata executions from an initial condition and a static rule, we introduce mappings which generate iteration rules from the cellular automaton itself. We show that these new automata contain disconnected regions which locally act like conventional automata, thus encapsulating multiple functions into one structure. Consequently, we have provided a new model for processes like cell differentiation. Finally, by studying the size of these regions, we provide additional evidence that the dynamics of self-reference may be critical to understanding the evolution of natural language. In particular, the rules of elementary cellular automata appear to be distributed in the same way as words in the corpus of a natural language.

show abstract

Natural language from function dynamics

Cited by 6 publications

References 10 publications

Transient Chaos in BERT

Transient Chaos in BERT

Momentum in Language Change

Self-Referencing Cellular Automata: A Model of the Evolution of Information Control in Biological Systems

Contact Info

Product

Resources

About