Am I a Resource-Poor Language? Data Sets, Embeddings, Models and Analysis for four different NLP Tasks in Telugu Language

Abstract

Due to the lack of a large annotated corpus, many resource-poor Indian languages struggle to reap the benefits of recent deep feature representations in Natural Language Processing (NLP). Moreover, adopting existing language models trained on large English corpora for Indian languages is often limited by data availability, rich morphological variation, syntax, and semantic differences. In this paper, we explore the traditional to recent efficient representations to overcome the challenges of low resource language, Telugu. In particular, our main objective is to mitigate the low-resource problem for Telugu. Overall, we present several contributions to a resource-poor language viz. Telugu. (i) a large annotated data (35,142 sentences in each task) for multiple NLP tasks such as sentiment analysis, emotion identification, hate-speech detection, and sarcasm detection, (ii) we create different lexicons for sentiment, emotion, and hate-speech for improving the efficiency of the models, (iii) pretrained word and sentence embeddings, and (iv) different pretrained language models for Telugu such as ELMo-Te , BERT-Te , RoBERTa-Te , ALBERT-Te , and DistilBERT-Te on a large Telugu corpus consisting of 80,15,588 sentences (16,37,408 sentences from Telugu Wikipedia and 63,78,180 sentences crawled from different Telugu websites). Further, we show that these representations significantly improve the performance of four NLP tasks and present the benchmark results for Telugu. We argue that our pretrained embeddings are competitive or better than the existing multilingual pretrained models: mBERT , XLM-R , and IndicBERT . Lastly, the fine-tuning of pretrained models show higher performance than linear probing results on four NLP tasks with following F1-scores: Sentiment (68.72), Emotion (58.04), Hate-Speech (64.27) and Sarcasm (77.93). We also experiment on publicly available Telugu datasets (Named Entity Recognition, Article Genre Classification, and Sentiment Analysis), find that our Telugu pretrained language models ( BERT-Te and RoBERTa-Te ) outperform the state-of-the-art system except for the sentiment task. We open-source our corpus, four different datasets, lexicons, embeddings, and code https://github.com/Cha14ran/DREAM-T. The pretrained Transformer models for Telugu are available at https://huggingface.co/ltrctelugu.

Sentiment Analysis in Low-Resource Settings: A Comprehensive Review of Approaches, Languages, and Data Sources

“…One-hot encoding [35], [74] CBOW [96], , [97], [64], [68], [89], [75], [104] Skip Gram [80], [62], [68], [89] Embedding [89], [83], [69], [70], [91], [84], [127], [81],…”

“…The result achieved 74.88% f1-score with the BERT model in the multi-tasks approach in aspect polarity classification. A study by [89] explored the use of four Telugu pre-trained word embeddings to solve the problem of Telugu being a low-resource language. One of the classifiers that were used for this task is LTSM.…”

“…Nonetheless, challenges persist, particularly in accurately representing all languages within tokenizer frameworks due to variations in language characters and other linguistic factors. Data cleaning [95], [35], [96], [49], [62], [65], [58], [68], [67], [89], [74], [75], [76], [77], [88], [94], [100], [103], [104], [107], [80], [106], [81], [108], [109], [82], [53], [110], [111], [83], [92], [113], [114], [115], [116], [117] Stemming [51], [67], [104], [108],…”

“…[80], [106], [53], [110], [83], [112], [113], [115], [116] Normalization [89], [77], [104], [80], [108], [83] Name Entity Reg (NER) [48], [108] N/A [64], [69], [70], [91], [84], [90], [101], [102], [127], [66] Table 9 Shows the analysis of the feature extraction method across reviewed studies in sentiment analysis for low-resource languages. The review reveals several trends.…”

“…In the domain of low-resource sentiment analysis, several challenges arise, primarily from the limited availability of digital resources [89,129]. The effectiveness of sentiment analysis is inherently linked to the accessibility of datasets, which traditionally abound for high-resource languages [48].…”