Emergence of order in random languages

Abstract

We consider languages generated by weighted context-free grammars. It is shown that the behavior of large texts is controlled by saddle-point equations for an appropriate generating function. We then consider ensembles of grammars, in particular the Random Language Model of [1]. This model is solved in the replicasymmetric ansatz, which is valid in the high-temperature, disordered phase. It is shown that in the phase in which languages carry information, the replica symmetry must be broken.Many complex systems have a generative, or linguistic, aspect. For example, protein structure is written in sequences of amino acids, a language of 20 different symbols. A large body of previous work has investigated the social aspect of linguistic systems, namely that different agents must find consensus regarding the meaning of symbols [2,3,4]. A complementary but necessary aspect of any linguistic system concerns the hidden structure within the sequences themselves, independent of communication. The most basic structural property is syntax: the rules that govern how symbols can be combined to create richer structures and thus carry information. In computer science and linguistics, generative grammar has proved to be a valuable formalism to describe syntax, in a generalized sense [5,6,7]. A generative grammar consists of an alphabet of hidden symbols, an alphabet of observable symbols, and a set of rules, which allow certain combinations of symbols to be replaced by others. From an initial start symbol S, one progressively applies the rules until only observable symbols remain; any sentence produced this way is said to be grammatical, and the set of all such sentences is called the language of the grammar. The sequence of rule applications is called a derivation. The Chomsky hierarchy distinguishes grammars based on the complexity of the grammatical rules. In this work, we restrict our attention to context-free grammars (CFGs), for which derivations are trees (Figure 1).There are many theoretical results on the capabilities of CFGs [7]. However, little is known about the statistical properties of large, typical grammars. Recently, there has been increasing interest in approaching the properties of syntax from the point of arXiv:1902.07516v2 [cond-mat.dis-nn]

Breakdown of random matrix universality in Markov models

“…These are related to syntactic structure in both natural and computer languages. A similar phase transition to that discussed here was found in [16,17], but without any interpretation in terms of random matrix theory. As discussed therein, the phase transition can be interpreted in terms of breaking of permutation sym-metry, an interpretation that also holds in the present case.…”

“…One can then look for phase transitions in the ensemble, with the aim of mapping out the general behaviors available within the class of model. This programme has been advocated as a general strategy for the study of complex systems [38], and has been attempted for constraint satisfaction problems [33], for language syntax [16,17], and for ecosystems [5], to give a few examples.…”

“…First, Hidden Markov models are the simplest class of probabilistic grammars, which are organized by the Chomsky hierarchy [7,10,17]. The next level of complexity in grammars are the context-free grammars, considered as an ensemble in [16,17]. These are related to syntactic structure in both natural and computer languages.…”

“…4c we estimate Ĩpred / log N ≈ 0.4 for the random model. Using (16) this implies H π / log N = log(2) × 0.4 + 0.6 = 0.88, which, again using ( 16), then implies I pred / log N = (0.88 − 0.26)/ log(2) = 0.89. This is remarkably close to the measured mean value 0.85.…”

Absence of Phase Transition in Random Language Model

“…Recently, DeGiuli, in their study on the typical properties of PCFG, introduced a model of PCFG averaged over grammars, called the Random Language Model (RLM) [13]. This model can be viewed as a statistical-mechanical model of random systems, which helps derive the free energy formula of the model using theoretical physics methods, similar to Feynman diagrams [14]. Numerical simulations [13] and statisticalmechanical analyses [14] of the model suggest that a phase transition occurs between ordered and random phases as the model parameters vary, demonstrating the emergence of order in language just as a child initially speaks incoherently, but later learns to speak grammatically correct language.…”