A New Methodology for Language Identification in Social Media Code-Mixed Text

Gupta, Yogesh; Raghuwanshi, Ghanshyam; Tripathi, Aprna

doi:10.1007/978-981-15-3383-9_22

Cited by 5 publications

(10 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Jamatia et al [54], the pre-trained word embeddings could improve the performance of the proposed code-mixed LID system. Another study by Shekhar et al [60] and Gupta et al [61] demonstrated that word embedding performed better than the character embedding technique. The word embeddings can identify language separation by detecting the word origin and mapping it to the correct language label.…”

Section: 2) Word Embeddingmentioning

confidence: 98%

“…We encountered three research by [11], [60], and [61] for the hybrid technique that combined the non-neural network and neural network techniques. They proposed similar architecture consisting of two modules: a multichannel neural network (MNN) and BLSTM-CRF.…”

Section: 1) Machine Learning Approachmentioning

confidence: 99%

“…Another work by [60] identified Hindi-English code-mixed text from social media platforms and gained the best F1 score of 93.97%. In [61], the best result was obtained by implementing 3-gram word embedding in their MNN-BLSTM-CRF architecture with an F1 score of 93.97%.…”

Section: 1) Machine Learning Approachmentioning

confidence: 99%

“…Another research by [15] combined character and word embedding representations by applying two LSTMs to train fixeddimensional representations from the embedding layers. In [60] and [61], character embedding was also employed with word embedding.…”

Section: 3) Character Embeddingmentioning

confidence: 99%

“…The ambiguity existed in several LID of code-mixed text studies, for example, Punjabi-English [16], Hindi-English [15,37,61], Malayalam-English [64], Bengali-English [37], Gujarati-English [37,59], Spanish-English [15,38,42,72], Dutch-English [20], Turkish-German & Spanish-Wixarika [22], Modern Standard Arabic-Arabic [42,72], Konkani-English [45], Swahili-English [40], English-Assamese-Hindi-Bengali [63], Sinhala-English [56]. The annotation of mixed languages becomes increasingly complicated when the languages are closely related [16].…”

Section: ) Ambiguitymentioning

confidence: 99%

See 4 more Smart Citations

A Systematic Review on Language Identification of Code-Mixed Text: Techniques, Data Availability, Challenges, and Framework Development

et al. 2022

View full text Add to dashboard Cite

The mix of native language with other languages (code-mixing) in social media has posed a severe challenge for language identification (LID) systems. It has encouraged research on code-mixed LID solutions. This study investigated the techniques, challenges, and dataset availability with corresponding quality criteria and developed a comprehensive framework for code-mixed LID. This study addressed four research issues to identify gaps and future work opportunities in tackling code-mixed LID challenges. Based on our analysis of reviewed studies, we outlined key points for future research in code-mixed LID. We demonstrated a taxonomy of applied techniques for code-mixed LID and highlighted the different technique variants. In code-mixed LID tasks, we discovered four significant challenges: ambiguity, lexical borrowing, non-standard words, and intra-word code-mixing. This systematic literature review recognised 32 code-mixed datasets available for LID. We proposed five features to describe the quality criteria dataset. The features are the number of instances or sentences, percentage of code-mixed types in the data, number of tokens, number of unique tokens, and average sentence length. Finally, we synthesised the methodologies and proposed a conceptual framework for subsequent studies through our literature analysis.

show abstract

Section: 2) Word Embeddingmentioning

confidence: 98%

Section: 1) Machine Learning Approachmentioning

confidence: 99%

Section: 1) Machine Learning Approachmentioning

confidence: 99%

Section: 3) Character Embeddingmentioning

confidence: 99%

Section: ) Ambiguitymentioning

confidence: 99%

See 3 more Smart Citations

A Systematic Review on Language Identification of Code-Mixed Text: Techniques, Data Availability, Challenges, and Framework Development

et al. 2022

View full text Add to dashboard Cite

show abstract

A Conceptual Framework for Malay-English Code-Switched Neural Machine Translation

Wong,

Huspi

2024

Lecture Notes on Data Engineering and Communications Technologies

View full text Add to dashboard Cite

TongueSwitcher: Fine-Grained Identification of German-English Code-Switching

Sterner,

Teufel

2023

Proceedings of the 6th Workshop on Computational Approaches to Linguistic Code-Switching

View full text Add to dashboard Cite

This paper contributes to German-English code-switching research. We provide the largest corpus of naturally occurring German-English code-switching, where English is included in German text, and two methods for code-switching identification. The first method is rule-based, using wordlists and morphological processing. We use this method to compile a corpus of 25.6M tweets employing German-English code-switching. In our second method, we continue pretraining of a neural language model on this corpus and classify tokens based on embeddings from this language model. Our systems establish SoTA on our new corpus and an existing German-English code-switching benchmark. In particular, we systematically study code-switching for language-ambiguous words which can only be resolved in context, and morphologically mixed words consisting of both English and German morphemes. We distribute both corpora and systems to the research community.

show abstract

A New Methodology for Language Identification in Social Media Code-Mixed Text

Cited by 5 publications

References 7 publications

A Systematic Review on Language Identification of Code-Mixed Text: Techniques, Data Availability, Challenges, and Framework Development

A Systematic Review on Language Identification of Code-Mixed Text: Techniques, Data Availability, Challenges, and Framework Development

A Conceptual Framework for Malay-English Code-Switched Neural Machine Translation

TongueSwitcher: Fine-Grained Identification of German-English Code-Switching

Contact Info

Product

Resources

About