Language Identification on Code-Mix Social Text

Kalita, Nayan Jyoti; Saharia, Navanath

doi:10.1007/978-981-10-6890-4_42

Cited by 5 publications

(13 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their work demonstrated that the SVM using word embedding obtained better results than Naïve Bayes and Convolutional Neural Network (CNN) with an F1 score of 90.61%. Kalita and Saharia [21] applied linear kernel SVM with Ngram and dictionary features to identify Assamese-English code-mixed language. They obtained 89.51% accuracy in word-level identification.…”

Section: 1) Machine Learning Approachmentioning

confidence: 99%

“…We found two different N-gram techniques being applied from the selected studies: word or token N-gram and character N-gram. Word or token N-gram has been used by [21,46,62]. In terms of word-level code-mixed LID task, the character N-gram is more popular than the word N-gram, especially for identifying the language in code-mixed script [15,16,58].…”

Section: 1) N-grammentioning

confidence: 99%

“…Piergallini et al [40] utilised English and Swahili dictionary features combined with the two other features: capitalisation and regular expression features. Kalita and Saharia [21] incorporated three different dictionaries with N-gram features. Bansal et al [16] used dictionaries as features to express the presence of words.…”

Section: 5) Dictionary Featuresmentioning

confidence: 99%

“…Detecting mixed-language text is a hot topic in natural language processing research. Most traditional language detectors do not identify mixed-language texts because identifying the language in code-mixed text requires different techniques from multiple languages applied to a document [21]. Moreover, code-mixed text that uses various levels combinations (sentence, clause, word, and sub-word) make LID a more complicated [15] than text expressed in one language.…”

Section: B (Rq2) What Are the Challenges In Lid Of Code-mixed Text?mentioning

confidence: 99%

“…Performing LID tasks at the document or sentence level are frequently inadequate for extracting critical information from the text [18]. Using language tags at the document or sentence level is insufficient, making language detector systems fail to detect correct language due to the mixed language in the sentences [19,21]. Therefore, researchers were motivated to shift their focus to word-level LID.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

A Systematic Review on Language Identification of Code-Mixed Text: Techniques, Data Availability, Challenges, and Framework Development

et al. 2022

View full text Add to dashboard Cite

The mix of native language with other languages (code-mixing) in social media has posed a severe challenge for language identification (LID) systems. It has encouraged research on code-mixed LID solutions. This study investigated the techniques, challenges, and dataset availability with corresponding quality criteria and developed a comprehensive framework for code-mixed LID. This study addressed four research issues to identify gaps and future work opportunities in tackling code-mixed LID challenges. Based on our analysis of reviewed studies, we outlined key points for future research in code-mixed LID. We demonstrated a taxonomy of applied techniques for code-mixed LID and highlighted the different technique variants. In code-mixed LID tasks, we discovered four significant challenges: ambiguity, lexical borrowing, non-standard words, and intra-word code-mixing. This systematic literature review recognised 32 code-mixed datasets available for LID. We proposed five features to describe the quality criteria dataset. The features are the number of instances or sentences, percentage of code-mixed types in the data, number of tokens, number of unique tokens, and average sentence length. Finally, we synthesised the methodologies and proposed a conceptual framework for subsequent studies through our literature analysis.

show abstract

Section: 1) Machine Learning Approachmentioning

confidence: 99%

Section: 1) N-grammentioning

confidence: 99%

Section: 5) Dictionary Featuresmentioning

confidence: 99%

Section: B (Rq2) What Are the Challenges In Lid Of Code-mixed Text?mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Systematic Review on Language Identification of Code-Mixed Text: Techniques, Data Availability, Challenges, and Framework Development

et al. 2022

View full text Add to dashboard Cite

show abstract

BERT Based Language Identification in Code-Mixed English-Assamese Social Media Text

Kalita,

Deka,

Chennareddy

et al. 2023

Algorithms for Intelligent Systems

View full text Add to dashboard Cite

Word Level Language Identification on Code-Mixed English-Bodo Text

Kalita

Agarwala

Das

2021

IOP Conf. Ser.: Mater. Sci. Eng.

View full text Add to dashboard Cite

Since social media has become an active part of one’s life, people express their views freely in mixed informal languages on such platforms. So, in a multi-lingual country like India, it becomes really difficult for conventional language detectors to identify such languages. This paper mainly aims to detect the language at word level where the code mixed text can be in English-Bodo-Assamese. The data for the same is collected from some related Facebook pages and various classification algorithms are used to predict and compare the accuracy with which the detection is done.

show abstract

Language Identification on Code-Mix Social Text

Cited by 5 publications

References 15 publications

A Systematic Review on Language Identification of Code-Mixed Text: Techniques, Data Availability, Challenges, and Framework Development

A Systematic Review on Language Identification of Code-Mixed Text: Techniques, Data Availability, Challenges, and Framework Development

BERT Based Language Identification in Code-Mixed English-Assamese Social Media Text

Word Level Language Identification on Code-Mixed English-Bodo Text

Contact Info

Product

Resources

About