Unsupervised Deep Language and Dialect Identification for Short Texts

Abstract

Automatic Language Identification (LI) or Dialect Identification (DI) of short texts of closely related languages or dialects, is one of the primary steps in many natural language processing pipelines. Language identification is considered a solved task in many cases; however, in the case of very closely related languages, or in an unsupervised scenario (where the languages are not known in advance), performance is still poor. In this paper, we propose the Unsupervised Deep Language and Dialect Identification (UDLDI) method, which can simultaneously learn sentence embeddings and cluster assignments from short texts. The UDLDI model understands the sentence constructions of languages by applying attention to character relations which helps to optimize the clustering of languages. We have performed our experiments on three shorttext datasets for different language families, each consisting of closely related languages or dialects, with very minimal training sets. Our experimental evaluations on these datasets have shown significant improvement over state-of-the-art unsupervised methods and our model has outperformed state-of-the-art LI and DI systems in supervised settings.

Weakly-supervised Deep Cognate Detection Framework for Low-Resourced Languages Using Morphological Knowledge of Closely-Related Languages

“…Character Encoding. The character level CNN layer generates word representation on n-gram (n ∈ {2,3,4,5,6}) characters, which helps to understand the representation of words on different sub-word levels (Goswami et al, 2020). A word's different sub-word level representations are achieved using a 1-dimensional CNN (Zhang et al, 2015).…”

“…where k is the number of classes. We train the unsupervised model based on the maximum likelihood clustering loss proposed by Goswami et al (2020), where they try to maximize the probability distribution function for each class and at the same time try to minimize the probability of all the datasets to be assigned to one class using Equation 7.…”

“…To alleviate the above challenges, in this paper, we propose a language-agnostic weakly-supervised cognate detection framework based on Siamese architecture with an iterative clustering approach (Xie et al, 2016) during back-propagation. Our encoder design is inspired by Goswami et al (2020), where they learn the n-gram character features of a sentence with attention. We introduce a positional encoder on n-gram features, which, in combination with the attention mechanism, learns sub-word representations of a word.…”

Optimizing the Size of Subword Vocabularies in Dialect Classification

“…On the side of CNNs, we evaluate two concrete architectures: Kim_CNN (Kim et al, 2016) and Zhang_CNN (Zhang et al, 2015), which are known to perform well on the task of text classification and are widely used. On the side of RNNs, we evaluate the architecture Lin_SA_BiLSTM (Lin et al, 2017), which has been shown to give good results on the task of dialect classifications (Goswami et al, 2020). We manipulated the tokenizers of these models using different granularity levels without changing the overall architecture.…”

Comparing Approaches to Dravidian Language Identification

“…akshara n-grams) work better than features based on bytes. Goswami et al (2020) experiment with supervised and unsupervised methods in dialect identification using, among others, a Dravidian data set containing Tamil, Telugu, Malayalam, and Kannada.…”