Dialectones: Finding statistically significant dialectal boundaries using Twitter data

Abstract

Abstract. Most NLP applications assume that a particular language is homogeneous in the regions where it is spoken. However, each language varies considerably throughout its geographical distribution. To make NLP sensitive to dialects, a reliable, representative and up-to-date source of information that quantitatively represents such geographical variation is necessary. However, some of the current approaches have disadvantages such as the need for parameters, ignoring the geographical coordinates in the analysis, and the use of linguistic alternations that presuppose the existence of specific dialectal varieties. Detection of "ecotones" is an analogous problem in the field of ecology that focuses on the identification of boundary areas in ecosystems instead of regions, facilitating the construction of statistical tests. We adapted the concept of "ecotone" to "dialectone" for the detection of dialectal boundary areas by using two non-parametric statistical tests: the Hilbert-Schmidt independence criterion (HSIC) and the Wilcoxon signed-rank. The proposed method was applied to a large corpus of Spanish tweets produced in 160 locations in Colombia through the analysis of unigram features. The resulting dialectones showed to be meaningful but difficult to compare against regions identified by other authors using classical dialectometry. We concluded that the automatic detection of dialectones is convenient alternative to classical methods in dialectometry and a potential source of information for automatic language applications.

Regionalized models for Spanish language variations based on Twitter

“…The study was made using a classification language model with good performance over multiple languages. In [8] the authors study Spanish language variations in Colombia. The analysis used uni-gram features, and the authors stated that it was challenging to compare Spanish variations against regions identified by other authors using classical dialectometry.…”

“…Both language and geographical information are crucial to know and understand the geographies of this online data, also the way some information related to economic, social, political, environmental trends could be used [7]. In contrast, it is not easy to accurately analyze variations of language using only classical dialectometry [8]; therefore, we aim for approaches more related to machine learning and natural language processing solutions to be able to handle very large datasets in our study.…”

El español de Colombia. Nueva propuesta de división dialectal

“…Realizado a partir de un corpus diferente, está el trabajo de Rodríguez-Díaz, Jiménez, Dueñas, Bonilla y Gelbukh (2018). El equipo investigador recolectó a través de la red social Twitter un corpus de 28 millones de trinos registrados en 237 localidades del territorio colombiano, y los procesó estadísticamente de acuerdo con diferentes medidas y pruebas.…”

“…Fuente: elaboración propia con base en los datosde Ávila et al (2015),Bonilla (2019),IGAC (2002),Mora et al (2004) y Rodríguez-Díaz et al (2018 …”

“…Los investigadores proponen el concepto de "dialectono", que conciben como el límite entre áreas dialectales adyacentes, caracterizado por el grado de interacción entre las variedades delimitadas (cf Rodríguez-Díaz et al, 2018)…”