Continuing Pre-trained Model with Multiple Training Strategies for Emotional Classification

Abstract

Emotion is the essential attribute of human beings. Perceiving and understanding emotions in a human-like manner is the most central part of developing emotional intelligence. This paper describes the contribution of the LingJing team's method to the Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis (WASSA) 2022 shared task on Emotion Classification. The participants are required to predict seven emotions from empathic responses to news or stories that caused harm to individuals, groups, or others. This paper describes the continual pre-training method for the masked language model (MLM) to enhance the DeBERTa pre-trained language model. Several training strategies are designed to further improve the final downstream performance including the data augmentation with the supervised transfer, child-tuning training, and the late fusion method. Extensive experiments on the emotional classification dataset show that the proposed method outperforms other state-of-the-art methods, demonstrating our method's effectiveness. Moreover, our submission ranked Top-1 with all metrics in the evaluation phase for the Emotion Classification task.

Late Fusion of Transformers for Sentiment Analysis of Code-Switched Data

“…Fine-tuning the models with small amounts of available labeled data produced much better results than earlier methods. Recent research includes the generation of synthetic code-switched data (for better fine-tuning), intermediate task training (Prasad et al, 2021), continued pre-training (Li et al, 2022) (masked Language modeling using code-switched texts), late fusion (Mundra et al, 2021) (using multiple pre-trained transformer-based models and combining their outputs using some logic to improve overall performance), and Data Augmentation (Mundra et al, 2021) -both Random Augmentation (RA) and Balanced Augmentation (BA), and custom attention models (Li et al, 2021). Other techniques are used to improve sentiment analysis results in code-switched texts specific to language pairs or dataset specifications.…”