A Probabilistic Generative Model of Linguistic Typology

Abstract

In the principles-and-parameters framework, the structural features of languages depend on parameters that may be toggled on or off, with a single parameter often dictating the status of multiple features. The implied covariance between features inspires our probabilisation of this line of linguistic inquirywe develop a generative model of language based on exponential-family matrix factorisation. By modelling all languages and features within the same architecture, we show how structural similarities between languages can be exploited to predict typological features with near-perfect accuracy, outperforming several baselines on the task of predicting heldout features. Furthermore, we show that language embeddings pre-trained on monolingual text allow for generalisation to unobserved languages. This finding has clear practical and also theoretical implications: the results confirm what linguists have hypothesised, i.e. that there are significant correlations between typological features and languages.

SIGTYP 2020 Shared Task: Prediction of Typological Features

“…Malaviya et al (2017); Murawaki (2017); Bjerva and Augenstein (2018a); Bjerva et al (2019c)), most such work does not take into account that both phylogenetic and geographic proximity should be controlled for. Languages which have shared common ancestry will often have similar typological features, hence training and evaluating on the same language family will tend to inflate the expected performance of the model (Bjerva et al, 2019a). In the data for this shared task, we make sure to control for both of these factors.…”

“…A survey of approaches to prediction of features is provided in Ponti et al (2019a, § 4.3). Some common approaches include prediction based on language representations learned as a by-product of model training (Östling and Tiedemann, 2017;Malaviya et al, 2017;Bjerva and Augenstein, 2018a;Bjerva et al, 2019c) and matrix factorisation (Murawaki, 2017;Bjerva et al, 2019a).…”

NEMO: Frequentist Inference Approach to Constrained Linguistic Typology Feature Prediction in SIGTYP 2020 Shared Task

“…In this ap proach, the languages are represented as random variables that are explained in terms of other lan guages related to each other through phylogenetic and spatial neighborhood graphs. Bjerva et al (2019) introduce a generative model inspired by the Chomskyan principlesandparameters frame work, drawing on the correlations between typo logical features of languages to tackle the novel task of typological collaborative filtering, a con cept borrowed from the area of recommender sys tems.…”

Real-Valued Logics for Typological Universals: Framework and Application

“…The availability of comparable treebanks -syntactically annotated corpora -for a growing number of typologically distinct languages (most prominently in the collaborative Universal Dependencies project (Nivre et al, 2016)) has led to a recent surge of interest in computational work aiming to detect systematic patterns in the grammatical systems of natural languages and/or to test hypotheses from theoretical work in language typology against empirical evidence. The treebank-based approach (Liu, 2010;Lochbihler, 2017;Gerdes et al, 2019;Bjerva et al, 2019c;Hahn et al, 2020) adds a more data-driven perspective to a strand of research in computational typology (Daumé and Campbell, 2007;Malaviya et al, 2017;Oncevay et al, 2019;Bjerva et al, 2019a;Bjerva et al, 2019b) that is based on carefully curated typological databases such as WALS 1 (Dryer and Haspelmath, 2013) or URIEL 2 .…”

“…A major focus has been on (a) detecting universals that have the form of an implication between two typological variables, and (b) predicting the value of unknown features in typological databases based on systematic patterns in attested grammatical systems. Graphical models have been widely used to calculate the strength of an implication (Daumé and Campbell, 2007;Lu, 2013;Bjerva et al, 2019b;Bjerva et al, 2019a). While this approach is suitable if one wants to marginalize out the influence of confounding variables, it also constrains the investigated universals to have the form of an implication consisting of one implicand and usually one (but possibly multiple) implicant(s).…”