RoBERTa-GRU: A Hybrid Deep Learning Model for Enhanced Sentiment Analysis

Abstract

This paper proposes a novel hybrid model for sentiment analysis. The model leverages the strengths of both the Transformer model, represented by the Robustly Optimized BERT Pretraining Approach (RoBERTa), and the Recurrent Neural Network, represented by Gated Recurrent Units (GRU). The RoBERTa model provides the capability to project the texts into a discriminative embedding space through its attention mechanism, while the GRU model captures the long-range dependencies of the embedding and addresses the vanishing gradients problem. To overcome the challenge of imbalanced datasets in sentiment analysis, this paper also proposes the use of data augmentation with word embeddings by over-sampling the minority classes. This enhances the representation capacity of the model, making it more robust and accurate in handling the sentiment classification task. The proposed RoBERTa-GRU model was evaluated on three widely used sentiment analysis datasets: IMDb, Sentiment140, and Twitter US Airline Sentiment. The results show that the model achieved an accuracy of 94.63% on IMDb, 89.59% on Sentiment140, and 91.52% on Twitter US Airline Sentiment. These results demonstrate the effectiveness of the proposed RoBERTa-GRU hybrid model in sentiment analysis.

Improving sentiment classification using a RoBERTa-based hybrid model

“…Among the ML techniques are SVM ( Rahat et al, 2019 ; Steinke et al, 2022 ), DT ( Steinke et al, 2022 ), and KNN ( Rahat et al, 2019 ). LSTM ( Jang et al, 2020 ; Umer et al, 2021 ) CNN-LSTM ( Jang et al, 2020 ; Umer et al, 2021 ), RoBERTa-LSTM ( Tan et al, 2022 ), and RoBERTa-GRU ( Tan et al, 2023 ) are a few examples of deep learning techniques. Table 4 compares all approaches with the suggested RoBERTa-(CNN-LSTM) model using the Twitter dataset.…”

“…Among the ML techniques are SVM (Rahat et al, 2019;Steinke et al, 2022), DT (Steinke et al, 2022), and KNN (Rahat et al, 2019). LSTM (Jang et al, 2020;Umer et al, 2021) CNN-LSTM (Jang et al, 2020;Umer et al, 2021), RoBERTa-LSTM (Tan et al, 2022), and RoBERTa-GRU (Tan et al, 2023) On the IMDB reviews dataset, the suggested model and other techniques are contrasted in Table 5. The values of F1-measure, precision, accuracy, and recall are the highest for our proposed model.…”

“…As shown in Table 1, sentiment analysis has been compared with other methods. Tan et al (2023) suggested a novel hybrid sentiment analysis model. The suggested model employs the Gated Recurrent Units (GRU) and Robustly Optimized BERT Pretraining Approach (RoBERTa) models.…”

Fake News Multi Class Detection Using Transformers and Gradient Boosting Ensemble

“…Contemporary research is gravitating towards hybrid models that combine deep learning's text analytic depth with machine learning's efficiency in numerical data interpretation, aiming for superior classification accuracy [6], [7]. Our study contributes to this trajectory by proposing a multi-class detection framework that unites BERT and Roberta's contextually aware models [8], [9], [10] with BiLSTM's sequential data handling [11], [12] and LightGBM's ensemble strengths. A thorough literature review was performed, and several other classification studies [13], [14], [15] were analyzed.…”

Academic assistance chatbot-a comprehensive NLP and deep learning-based approaches

“…The dataset was then split into training, validation, and testing sets. The preprocessed dataset was then formatted into a compatible format to be fed to the RoBERTa model which was done by concatenating and adding special tokens [30]. The typical format for RoBERTa input is set to be as: single sequence: <s> X </s> pair of sequences: <s> A </s></s> B </s>.…”