A Kernel Independence Test for Geographical Language Variation

Abstract

Quantifying the degree of spatial dependence for linguistic variables is a key task for analyzing dialectal variation. However, existing approaches have important drawbacks. First, they are based on parametric models of dependence, which limits their power in cases where the underlying parametric assumptions are violated. Second, they are not applicable to all types of linguistic data: some approaches apply only to frequencies, others to boolean indicators of whether a linguistic variable is present. We present a new method for measuring geographical language variation, which solves both of these problems. Our approach builds on Reproducing Kernel Hilbert space (RKHS) representations for nonparametric statistics, and takes the form of a test statistic that is computed from pairs of individual geotagged observations without aggregation into predefined geographical bins. We compare this test with prior work using synthetic data as well as a diverse set of real datasets: a corpus of Dutch tweets, a Dutch syntactic atlas, and a dataset of letters to the editor in North American newspapers. Our proposed test is shown to support robust inferences across a broad range of scenarios and types of data.

A Neural Model for User Geolocation and Lexical Dialectology

“…The traditional approach to dialectology is to find the geographical distribution of known lexical alternatives (e.g. you, yall and yinz: (Labov et al, 2005;Nerbonne et al, 2008;Gonçalves and Sánchez, 2014;Doyle, 2014;Huang et al, 2015;Nguyen and Eisenstein, 2016)), the shortcoming of which is that the alternative lexical variables must be known beforehand. There have also been attempts to automatically identify such words from geotagged documents (Eisenstein et al, 2010;Ahmed et al, 2013;Eisenstein, 2015).…”

Dialectones: Finding statistically significant dialectal boundaries using Twitter data

“…For that, the Hilbert-Schimidt independence criterion (HSIC) has been used following the recent findings of Nguyen & Eisenstein (2017) [13], who showed that HSIC outperformed other known spatial autocorrelation tests (i.e. Moran's I, Join Count Analysis, and the Mantel test) in the task of detecting geographical language variation.…”

“…In contrast, filtering the entire vocabulary throughout the entire corpus provide a larger set of significant unbiased features. Moreover, as Nguyen & Eisenstein showed [13], the HSIC test is a better alternative for Moran's I when using linguistics variables.…”

“…Principal Component Analysis, PCA); and 5) visualizing the found groups as regions in a geographical map. This methodology has several known issues that have been noted the in the recent literature [12,4,13]. These and other issues are discussed bellow.…”

Dialectones: Finding Statistically Significant Dialectal Boundaries Using Twitter Data

“…Principal Component Analysis, PCA); and 5) visualizing the found groups as regions in a geographical map. This methodology has several known issues that have been noted the in the recent literature [12,4,13].…”