Adapter-based fine-tuning of pre-trained multilingual language models for code-mixed and code-switched text classification

Rathnayake, Himashi; Sumanapala, Janani; Rukshani, Raveesha; Ranathunga, Surangika

doi:10.1007/s10115-022-01698-1

Cited by 13 publications

(14 citation statements)

References 26 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Adapters can be generally categorized into two categories: task adapters, which learn task-specific representations, and language adapters, which learn languagespecific representations [27]. Typically, language adapters are used in conjunction with task adapters [3,28]. Extensive research has been conducted on adapters as a parameter-efficient fine-tuning method for various tasks.…”

Section: Adapter-based Fine-tuning Of Plmsmentioning

confidence: 99%

“…Despite exhibiting success over full fine-tuning for monolingual text, prompt-based learning of PLMs with CMCS data for downstream tasks has not been explored. In the context of CMCS data, we are only aware of full-fine-tuning PLMs [3,16]. Given that prompt-based learning relies on textual prompts, the design of such prompts for CMCS text is an open question.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Use of Prompt-Based Learning for Code-Mixed and Code-Switched Text Classification

Udawatta,

Udayangana,

Gamage

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Code-mixing and code-switching (CMCS) are prevalent phenomena observed in social media conversations and various other modes of communication. When developing Natural Language Processing (NLP) systems such as sentiment analysers and hate-speech detectors that operate on this social media data, CMCS text poses challenges. Recent studies have demonstrated that prompt-based learning of pre-trained language models (PLMs) outperforms full fine-tuning of PLMs across various NLP tasks. Despite the growing interest in CMCS text classification, the effectiveness of prompt-based learning for the task remains unexplored. Our study endeavours to bridge this gap by examining the impact of prompt-based learning on CMCS text classification. We discern that the performance in CMCS text classification is significantly influenced by the inclusion of multiple scripts and the intensity of code-mixing. In response, we introduce a novel method, Dynamic+AdapterPrompt, which employs distinct models for each script, integrated with adapters. While DynamicPrompt captures the script-specific representation of CMCS text, AdapterPrompt emphasizes capturing the task-oriented functionality. Our experiments span across Sinhala-English, Kannada-English, and Hindi-English datasets, encompassing sentiment classification, hate-speech detection, and humour detection tasks. The outcomes indicate that our proposed method outperforms strong fine-tuning baselines and basic prompting strategies.

show abstract

Section: Adapter-based Fine-tuning Of Plmsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Use of Prompt-Based Learning for Code-Mixed and Code-Switched Text Classification

Udawatta,

Udayangana,

Gamage

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Adapter-BERT outperforms fine-tuned BERT in terms of performance. Figure 2 illustrates the architecture of adapter-BERT 17,18 .…”

Section: Proposed Systemmentioning

confidence: 99%

“…Out of all these models, hybrid deep learning model CNN + BiLSTM works well to perform sentiment analysis with an accuracy of 66%. In 18 , aspect based sentiment analysis known as SentiPrompt which utilizes sentiment knowledge enhanced prompts to tune the language model. This methodology is used for triplet extraction, pair extraction and aspect term extraction.…”

mentioning

confidence: 99%

Deep learning based sentiment analysis and offensive language identification on multilingual code-mixed data

Shanmugavadivel¹,

Sathishkumar

Raja³

et al. 2022

Sci Rep

View full text Add to dashboard Cite

Sentiment analysis is a process in Natural Language Processing that involves detecting and classifying emotions in texts. The emotion is focused on a specific thing, an object, an incident, or an individual. Although some tasks are concerned with detecting the existence of emotion in text, others are concerned with finding the polarities of the text, which is classified as positive, negative, or neutral. The task of determining whether a comment contains inappropriate text that affects either individual or group is called offensive language identification. The existing research has concentrated more on sentiment analysis and offensive language identification in a monolingual data set than code-mixed data. Code-mixed data is framed by combining words and phrases from two or more distinct languages in a single text. It is quite challenging to identify emotion or offensive terms in the comments since noise exists in code-mixed data. The majority of advancements in hostile language detection and sentiment analysis are made on monolingual data for languages with high resource requirements. The proposed system attempts to perform both sentiment analysis and offensive language identification for low resource code-mixed data in Tamil and English using machine learning, deep learning and pre-trained models like BERT, RoBERTa and adapter-BERT. The dataset utilized for this research work is taken from a shared task on Multi task learning DravidianLangTech@ACL2022. Another challenge addressed by this work is the extraction of semantically meaningful information from code-mixed data using word embedding. The result represents an adapter-BERT model gives a better accuracy of 65% for sentiment analysis and 79% for offensive language identification when compared with other trained models.

show abstract

“…Through the study of news text classification algorithms, it was found that traditional machine learning methods are prone to losing useful semantic feature information in the process of text representation, while using models such as Word2Vec and glove for text representation and then sharing text contextual semantic information by training neural network models can learn more vector representations as features, which is significantly better than traditional machine in terms of classification accuracy learning methods [7][8]. However, models such as Word2Vec cannot solve the problem of multiple meanings of words, especially in the face of the sparse features and context-dependent nature of news headlines, and there are still many semantic problems to be solved [9][10].…”

Section: Key Issues In News Text Classificationmentioning

confidence: 99%

Machine Learning Based News Text Classification

2020

View full text Add to dashboard Cite

With the rapid development of the Internet era, the explosive growth of news data volume and the lack of effective management are gradually becoming serious problems, and it is increasingly difficult for readers to obtain valuable information quickly. How to quickly search for valuable information from the large amount of news text information is a meaningful task in text classification. Existing research methods still have some problems, such as directly combining the headline with the text, thus neglecting the importance of the headline, and the single model of classification, which leads to low classification results. For this reason, the main objective of this paper is to investigate the classification of news texts based on machine learning. This paper examines the current state of deep learning-based text classification and, in combination with the characteristics of news texts, chooses a machine learning-based text classification method to further explore and study news texts. Through the response time and core algorithm accuracy test of this system, the system better reflects the excellent performance of the system and meets the actual performance requirements of the system. The proposed news system can not only process system requests quickly, but also has excellent accuracy rate, which can better assist users to filter information and improve the user's experience of reading news.

show abstract

Adapter-based fine-tuning of pre-trained multilingual language models for code-mixed and code-switched text classification

Cited by 13 publications

References 26 publications

Use of Prompt-Based Learning for Code-Mixed and Code-Switched Text Classification

Use of Prompt-Based Learning for Code-Mixed and Code-Switched Text Classification

Deep learning based sentiment analysis and offensive language identification on multilingual code-mixed data

Machine Learning Based News Text Classification

Contact Info

Product

Resources

About