Contextual Hate Speech Detection in Code Mixed Text using Transformer Based Approaches

Ravindra, Nayak,; Joshi, Raviraj

doi:10.48550/arxiv.2110.09338

Cited by 2 publications

(2 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we will try to mainly discuss previous attempts in the creation of code-mixed datasets. User-generated content is the main source of codemixed data, and preprocessing is necessary for tasks like profanity hate speech [20,3,11,22,17], sentiment analysis, etc. Various attempts of scraping have been done before for the initial set of code-mixed data and later augmented synthetically using equivalence constraint theory [19], semi-supervised learning [9] and rule-based language-pair approaches [26].…”

Section: Related Workmentioning

confidence: 99%

L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models

Ravindra¹,

Joshi²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Code-switching occurs when more than one language is mixed in a given sentence or a conversation. This phenomenon is more prominent on social media platforms and its adoption is increasing over time. Therefore code-mixed NLP has been extensively studied in the literature. As pre-trained transformer-based architectures are gaining popularity, we observe that real code-mixing data are scarce to pre-train large language models. We present L3Cube-HingCorpus, the first large-scale real Hindi-English code mixed data in a Roman script. It consists of 52.93M sentences and 1.04B tokens, scraped from Twitter. We further present HingBERT, HingMBERT, HingRoBERTa, and HingGPT. The BERT models have been pre-trained on codemixed HingCorpus using masked language modelling objectives. We show the effectiveness of these BERT models on the subsequent downstream tasks like code-mixed sentiment analysis, POS tagging, NER, and LID from the GLUECoS benchmark. The HingGPT is a GPT2 based generative transformer model capable of generating full tweets. We also release L3Cube-HingLID Corpus, the largest code-mixed Hindi-English language identification(LID) dataset and HingBERT-LID, a production-quality LID model to facilitate capturing of more code-mixed data using the process outlined in this work. The dataset and models are available at https://github.com/l3cubepune/code-mixed-nlp.

show abstract

Section: Related Workmentioning

confidence: 99%

L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models

Ravindra¹,

Joshi²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The state-of-the-art hostile text detection methods are available for English language. To enhance the research in low resource Indian languages, we have studied various methods which can detect hostile posts in Hindi (1, 2, 6-8, 13, 14), Marathi (7,11,12), Bengali (12), Saudi (4), Roman Urdu (15,16), Tamil (17) and codemixed language (7,(18)(19)(20)(21)(22). Figure 1 shows basic approaches used for hostile post-detection.…”

Section: Introductionmentioning

confidence: 99%

Investigating Hostile Post Detection in Gujarati: A Machine Learning Approach

Rameshbhai,

Rana

2024

IJE

View full text Add to dashboard Cite

Hostile post on social media is a crucial issue for individuals, governments and organizations. There is a critical need for an automated system that can investigate and identify hostile posts from large-scale data. In India, Gujarati is the sixth most spoken language. In this work, we have constructed a major hostile post dataset in the Gujarati language. The data are collected from Twitter, Instagram and Facebook. Our dataset consists of 1,51,000 distinct comments having 10,000 manually annotated posts. These posts are labeled into the Hostile and Non-Hostile categories. We have used the dataset in two ways: (i) Original Gujarati Text Data and (ii) English data translated from Gujarati text. We have also checked the performance of pre-processing and without pre-processing data by removing extra symbols and substituting emoji descriptions in the text. We have conducted experiments using machine learning models based on supervised learning such as Support Vector Machine, Decision Tree, Random Forest, Gaussian Naive-Bayes, Logistic Regression, K-Nearest Neighbor and unsupervised learning based model such as k-means clustering. We have evaluated performance of these models for Bag-of-Words and TF-IDF feature extraction methods. It is observed that classification using TF-IDF features is efficient. Among these methods Logistic regression outperforms with an Accuracy of 0.68 and F1-score of 0.67. The purpose of this research is to create a benchmark dataset and provide baseline results for detecting hostile posts in Gujarati Language.

show abstract

Contextual Hate Speech Detection in Code Mixed Text using Transformer Based Approaches

Cited by 2 publications

References 21 publications

L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models

L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models

Investigating Hostile Post Detection in Gujarati: A Machine Learning Approach

Contact Info

Product

Resources

About