Indian Language Identification for Short Text

Bhaskaran, Sreebha; Paul, Geetika; Gupta, Deepa; Amudha, J.

doi:10.1007/978-981-15-1275-9_5

Cited by 11 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Language recognition tools (such as langid.py [ 8 ]) solve this problem and use different classification algorithms to solve the problem at the sentence level. Many methods have been utilized in [ 9 – 12 ] to handle the problem of classifying code-mixing by using different frameworks, such as n -gram [ 13 ], Malayalam-English used Bi-LSTM and Hindi-English used KNN in [ 14 , 15 ], parts of speech (POS) [ 16 ] on multiple languages pairs, hidden Markov model [ 17 ], combined Support Vector Machine and CRFs [ 2 ] applied on code-mixed languages pairs such as Spanish-English [ 18 ], Dutch-Turkish [ 19 ], Maltese-English [ 20 ], Romanized Arabic Moroccan (Darija), French-English [ 21 ], current standard Egyptian-Arabic dialect [ 22 ], English-Mandarin [ 23 , 24 ], and English-Malay [ 25 ]. Balazevic et al in [ 26 ] presented the integration of user-specific information to enhance the recognition of Twitter dataset in 16 languages.…”

Section: Related Workmentioning

confidence: 99%

Mixed Script Identification Using Automated DNN Hyperparameter Optimization

Yasir

Chen

Khatoon

et al. 2021

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Mixed script identification is a hindrance for automated natural language processing systems. Mixing cursive scripts of different languages is a challenge because NLP methods like POS tagging and word sense disambiguation suffer from noisy text. This study tackles the challenge of mixed script identification for mixed-code dataset consisting of Roman Urdu, Hindi, Saraiki, Bengali, and English. The language identification model is trained using word vectorization and RNN variants. Moreover, through experimental investigation, different architectures are optimized for the task associated with Long Short-Term Memory (LSTM), Bidirectional LSTM, Gated Recurrent Unit (GRU), and Bidirectional Gated Recurrent Unit (Bi-GRU). Experimentation achieved the highest accuracy of 90.17 for Bi-GRU, applying learned word class features along with embedding with GloVe. Moreover, this study addresses the issues related to multilingual environments, such as Roman words merged with English characters, generative spellings, and phonetic typing.

show abstract

Section: Related Workmentioning

confidence: 99%

Mixed Script Identification Using Automated DNN Hyperparameter Optimization

Yasir

Chen

Khatoon

et al. 2021

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…Firstly, code-mixed data involves switching between languages, where different languages are used interchangeably within a sentence or word. These language switches may follow speci c patterns or be in uenced by contextual factors (Withanage et al 2020;Attia et al 2019) Secondly, code-mixed text often incorporates non-standard spellings, abbreviations, acronyms, emoticons, hashtags, and other informal elements commonly found in social media conversations (Bhaskaran et al 2020). These linguistic phenomena pose challenges to existing POS tagging techniques, which heavily rely on formal language patterns and resources.…”

Section: Introductionmentioning

confidence: 99%

A Comprehensive Survey of Techniques Used for Part-of-Speech Tagging of Code-Mixed Social Media Text

Sunita,

Kumar,

Neetika

2023

Preprint

View full text Add to dashboard Cite

Part-of-speech tagging faces unique difficulties when dealing with code-mixed social media text, which combines multiple languages in informal content created by users. In India, many web users employ a mixture of regional languages and English on platforms like Facebook, Instagram, and WhatsApp to express their messages and emotions. Text derived from social media is used in a variety of applications, such as speech recognition, machine learning, information retrieval, question answering, sentiment analysis, and named entity recognition. Due to training with monolingual texts, natural language processing tools such as part-of-speech taggers and parsers don't perform well. Assigning grammatical labels to individual words (such as verbs, adjectives, and nouns) is a critical task in natural language processing. This review paper extensively surveys the existing literature on part-of-speech tagging specifically developed for Indian and Foreign code-mixed social media text. We examine and categorize the approaches utilized in prior studies, taking into account the diverse techniques and methodologies employed to handle the complexities of code-mixed data. These approaches encompass rule-based methods, statistical models, and deep learning techniques such as recurrent neural networks and transformers. To enable comprehensive analysis, we compare and evaluate the performance of various state-of-the-art code-mixed part-of-speech taggers using benchmark datasets. We discuss the evaluation metrics used in these studies. Lastly, we explore the challenges introduced by noisy and informal language commonly found in code-mixed social media text. This review paper serves as a valuable resource for researchers and practitioners seeking to understand the current state of the art in code-mixed part-of-speech tagging for social media text. It offers insights into the strengths and weaknesses of existing approaches, identifies research gaps, and proposes potential avenues for future research to advance the field.

show abstract

“…BharatBhasaNet marks a milestone in LID technology. As digital technologies, such as translation, ASR, and conversational interfaces, grow in importance [10], [12], [13], BharatBhasaNet emerges as a vital tool in creating resources for less commonly spoken languages [11]. It effectively navigates challenges such as web data noise, sparse datasets [12], and similarities among high-resource languages [14].…”

mentioning

confidence: 99%

BharatBhasaNet-A Unified Framework to Identify Indian Code Mix Languages

Dey,

Thakur,

Kandwal

et al. 2024

IEEE Access

View full text Add to dashboard Cite

In the rapidly globalizing digital communication sphere, the imperative for advanced multilingual text recognition and identification is increasingly evident. Contrasting the previous works, which were predominantly constrained to 2-3 languages, this paper explores the rich linguistic diversity of India, addressing challenges in automated language processing for 12 languages. BharatBhasaNet, our comprehensive Language Identification (LID) framework, integrates an extensive dataset covering these 12 Indian languages in both native-script and romanized forms, derived from INDICCORP [30], Bhasha-Abhijnaanam [20], and Aksharantar [32] datasets by AI4Bharat a . The framework accommodates two models, Roberta-native and Roberta-Romanized, based on attention mechanism and transformer architecture. With its exceptional accuracy of 99.54% in native script and 60.90% in Romanized text, BharatBhasaNet significantly advances language identification, providing broader language coverage than existing LIDs. It excels in interpreting code-mixed sentences , unveiling crucial accuracy patterns related to sentence length, word span, and complexity in multilingual contexts. The framework underwent rigorous testing using a real-time dataset from the National Informatics Center (NIC), achieving an accuracy rate of 92.67%. Overcoming challenges like limited training data and distinguishing similar languages, BharatBhasaNet marks a significant leap in Romanized text identification within diverse linguistic landscapes.a https://ai4bharat.iitm.ac.in

show abstract

Indian Language Identification for Short Text

Cited by 11 publications

References 12 publications

Mixed Script Identification Using Automated DNN Hyperparameter Optimization

Mixed Script Identification Using Automated DNN Hyperparameter Optimization

A Comprehensive Survey of Techniques Used for Part-of-Speech Tagging of Code-Mixed Social Media Text

BharatBhasaNet-A Unified Framework to Identify Indian Code Mix Languages

Contact Info

Product

Resources

About