SwitchNet: Learning to switch for word-level language identification in code-mixed social media text

Abstract

Word-level language identification is an essential prerequisite for extracting useful information from code-mixed social media content. Previous studies in word-level language identification show two important observations. First, the local context is an important indicator of the language of a word when a word is valid in multiple languages. Second, considering the word in isolation from its context leads to more effective language classification when a word is borrowed or embedded into sentences of other languages. In this paper, we propose a framework for language identification that makes use of a dynamic switching mechanism for effective language classification of both words that are borrowed or embedded from other languages as well as words that are valid in multiple languages. For a given input, the proposed switching mechanism makes a dynamic decision to bias its prediction either towards the prediction obtained by the contextual information or that obtained by the word in isolation. In contrast to existing studies that rely upon large amounts of annotated data for robust performance in a multilingual environment, the proposed approach uses minimal annotated resources and no external resources, making it easily extendible to newer languages. Evaluation over a corpus of transliterated Facebook comments shows that the proposed approach outperforms its baseline counterparts: classification based on the contextual information, classification based on the word in isolation, as well as an ensemble of the two classifiers.

A Systematic Review on Language Identification of Code-Mixed Text: Techniques, Data Availability, Challenges, and Framework Development

“…The convolution layer in CNN building blocks extracts text features by applying a convolutional filter or kernel to each window in the sequence of text. Sarma et al [63] experimented with CNN and BLSTM, and CNN has shown the best performance among the other techniques with an F1 score of 91.03%. Some studies combined two ANN modules, CNN and LSTM or CNN and BLSTM, in their neural network architecture.…”

“…The ambiguity existed in several LID of code-mixed text studies, for example, Punjabi-English [16], Hindi-English [15,37,61], Malayalam-English [64], Bengali-English [37], Gujarati-English [37,59], Spanish-English [15,38,42,72], Dutch-English [20], Turkish-German & Spanish-Wixarika [22], Modern Standard Arabic-Arabic [42,72], Konkani-English [45], Swahili-English [40], English-Assamese-Hindi-Bengali [63], Sinhala-English [56]. The annotation of mixed languages becomes increasingly complicated when the languages are closely related [16].…”

“…Another challenge related to ambiguity is lexical borrowing [63] Lexical borrowing is defined as transferring or copying a particular lexical item from one language to the lexicon of another language [20,73]. The LID system may have no difficulty distinguishing the words to the correct language tag for words that have phonetic similarities but have different spelling.…”

“…Austronesian & Germanic Indonesian-English [7] Malay-English [57] Dravidian & Germanic Kannada-English [44] Malayalam-English [47,64] Tamil-English [47,51] Telugu-English [50] Germanic & Trans Eurasian Dutch-Turkish [46] Germanic & Germanic Dutch-English [20,46,49] Dutch-Limburgish [43] German-English [46] Indo-Aryan & Germanic Assamese-English [21] Assamese-Hindi-Bengali-English [18,63] Bengali-English [11,27,54] Bengali-Hindi-English [54] Gujarati-English [59] Gujarati-Hindi-English [58] Hindi-English [11,15,37,48,52,54,60,61,65] Konkani-English [45] Punjabi-English [16] Sinhala-English [3,56] Italic & Germanic French-English [46] French-Italian-Spanish-English [55] Portuguese-English [46] Spanish-English [15,38,41,46,49] Italic & American Spanish-Wixarika…”

On-Device Major Indian Language Identification Classifiers to Run on Low Resource Devices

“…Neelakshi Sarma et al have developed a framework for language identification which is capable of recognizing words that are borrowed from other languages and used in multiple languages and predict the language. This framework also considers the context of the sentence [11].…”