Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks

Zixuan, Ke,; Xu, Huibin; Liu, Bing

doi:10.48550/arxiv.2112.03271

Cited by 3 publications

(9 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…BERT can take one or two sentences as input and differentiate them using the special token [SEP]. The [CLS] token, which is unique to classification tasks, always appears at the beginning of the text 17 .…”

Section: Proposed Systemmentioning

confidence: 99%

“…Adapter-BERT outperforms fine-tuned BERT in terms of performance. Figure 2 illustrates the architecture of adapter-BERT 17,18 .…”

Section: Proposed Systemmentioning

confidence: 99%

“…Sentiment analysis is performed on Tamil code-mixed data by capturing local and global features using machine learning, deep learning, transfer learning and hybrid models 17 . Out of all these models, hybrid deep learning model CNN + BiLSTM works well to perform sentiment analysis with an accuracy of 66%.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Deep learning based sentiment analysis and offensive language identification on multilingual code-mixed data

Shanmugavadivel¹,

Sathishkumar

Raja³

et al. 2022

Sci Rep

View full text Add to dashboard Cite

Sentiment analysis is a process in Natural Language Processing that involves detecting and classifying emotions in texts. The emotion is focused on a specific thing, an object, an incident, or an individual. Although some tasks are concerned with detecting the existence of emotion in text, others are concerned with finding the polarities of the text, which is classified as positive, negative, or neutral. The task of determining whether a comment contains inappropriate text that affects either individual or group is called offensive language identification. The existing research has concentrated more on sentiment analysis and offensive language identification in a monolingual data set than code-mixed data. Code-mixed data is framed by combining words and phrases from two or more distinct languages in a single text. It is quite challenging to identify emotion or offensive terms in the comments since noise exists in code-mixed data. The majority of advancements in hostile language detection and sentiment analysis are made on monolingual data for languages with high resource requirements. The proposed system attempts to perform both sentiment analysis and offensive language identification for low resource code-mixed data in Tamil and English using machine learning, deep learning and pre-trained models like BERT, RoBERTa and adapter-BERT. The dataset utilized for this research work is taken from a shared task on Multi task learning DravidianLangTech@ACL2022. Another challenge addressed by this work is the extraction of semantically meaningful information from code-mixed data using word embedding. The result represents an adapter-BERT model gives a better accuracy of 65% for sentiment analysis and 79% for offensive language identification when compared with other trained models.

show abstract

Section: Proposed Systemmentioning

confidence: 99%

“…Adapter-BERT outperforms fine-tuned BERT in terms of performance. Figure 2 illustrates the architecture of adapter-BERT 17,18 .…”

Section: Proposed Systemmentioning

confidence: 99%

See 1 more Smart Citation

Deep learning based sentiment analysis and offensive language identification on multilingual code-mixed data

Shanmugavadivel¹,

Sathishkumar

Raja³

et al. 2022

Sci Rep

View full text Add to dashboard Cite

show abstract

“…B-CL replaces adapters with CLA, containing a knowledge-sharing module (KSM) and taskspecific module (TSM), both with skip-connections. Image and modified caption are taken from (Ke et al, 2021b).…”

Section: Model Updating Challengesmentioning

confidence: 99%

“…For example, if we consider the BERT layer in Figure 3, we can observe that the adapter layers for the MTL approach in Figure 3(A) are in the same positions as the CL adapters in Figure 3(B) (Ke et al, 2021b). To switch from MTL to CL we only need to change adapters.…”

Section: Model Updating Challengesmentioning

confidence: 99%

Speeding Up Transformer Training By Using Dataset Subsampling - An Exploratory Analysis

Torbarina¹,

Mihelčić²,

Šarlija³

et al. 2021

Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing

View full text Add to dashboard Cite

Transformer-based models have greatly advanced the progress in the field of the natural language processing and while they achieve state-of-the-art results on a wide range of tasks, they are cumbersome in parameter size. Subsequently, even when pre-trained transformer models are used for fine-tuning on a given task, if the dataset is large, it may still not be feasible to fine-tune the model within a reasonable time. For this reason, we empirically test 8 subsampling methods for reducing the dataset size on text classification task and report the trade-off between metric score and training time. 7 out of 8 methods are simple methods, while the last one is CRAIG, a method for coreset construction for data-efficient model training. We obtain the best result with the CRAIG method, offering an average decrease of 0.03 points in f-score on test set while speeding up the training time on average by 63.93%, relative to the score and time obtained by using the full dataset. Lastly, we show the trade-off between speed and performance for all sampling methods on three different datasets.

show abstract

Contrastive Learning for Boosting Knowledge Transfer in Task-Incremental Continual Learning of Aspect Sentiment Classification Tasks

Dang,

Thi,

et al. 2024

JCSCE

View full text Add to dashboard Cite

Continual learning (CL) aims to learn a sequence of tasks, with task datasets emergingincrementally over time, and without a predetermined number of tasks. CL models strive to achievetwo primary objectives: preventing catastrophic forgetting and facilitating knowledge transfer between tasks. Catastrophic forgetting refers to the sharp decline in the performance of CL modelson previously learned tasks as new ones are learned. Knowledge transfer, which leverages acquiredknowledge from previous tasks, empowers the CL model to adeptly tackle new tasks. However, onlya few CL models proposed by far successfully achieve those two objectives simultaneously. In thispaper, we present a task-incremental CL based model that leverages a pre-trained language model(i.e., BERT) with injected CL-plugins to mitigate catastrophic forgetting in continual learning. Additionally, we propose the utilization of two contrastive learning-based losses, namely contrastiveensemble distillation (CED) and contrastive supervised learning of the current task (CSC) losses,to enhance our model’s performance. The CED loss improves the knowledge transferability of ourcontinual learning model, while the CSC loss enhances its performance for the current learning task.Experimental results on benchmark datasets demonstrate that our proposed model outperforms allexisting continual learning models in the task-incremental learning setting for continual aspect sentiment classification.Keywords: Continual Learning, Contrastive Learning, Aspect-Sentiment Classification.

show abstract

Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks

Cited by 3 publications

References 7 publications

Deep learning based sentiment analysis and offensive language identification on multilingual code-mixed data

Deep learning based sentiment analysis and offensive language identification on multilingual code-mixed data

Speeding Up Transformer Training By Using Dataset Subsampling - An Exploratory Analysis

Contrastive Learning for Boosting Knowledge Transfer in Task-Incremental Continual Learning of Aspect Sentiment Classification Tasks

Contact Info

Product

Resources

About