Attention-Based CNN and Bi-LSTM Model Based on TF-IDF and GloVe Word Embedding for Sentiment Analysis

Abstract

Sentiment analysis (SA) detects people’s opinions from text engaging natural language processing (NLP) techniques. Recent research has shown that deep learning models, i.e., Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Transformer-based provide promising results for recognizing sentiment. Nonetheless, CNN has the advantage of extracting high-level features by using convolutional and max-pooling layers; it cannot efficiently learn a sequence of correlations. At the same time, Bidirectional RNN uses two RNN directions to improve extracting long-term dependencies. However, it cannot extract local features in parallel, and Transformer-based like Bidirectional Encoder Representations from Transformers (BERT) are the computational resources needed to fine-tune, facing an overfitting problem on small datasets. This paper proposes a novel attention-based model that utilizes CNNs with LSTM (named ACL-SA). First, it applies a preprocessor to enhance the data quality and employ term frequency-inverse document frequency (TF-IDF) feature weighting and pre-trained Glove word embedding approaches to extract meaningful information from textual data. In addition, it utilizes CNN’s max-pooling to extract contextual features and reduce feature dimensionality. Moreover, it uses an integrated bidirectional LSTM to capture long-term dependencies. Furthermore, it applies the attention mechanism at the CNN’s output layer to emphasize each word’s attention level. To avoid overfitting, the Guasiannoise and GuasianDroupout are adopted as regularization. The model’s robustness is evaluated on four English standard datasets, i.e., Sentiment140, US-airline, Sentiment140-MV, SA4A with various performance matrices, and compared efficiency with existing baseline models and approaches. The experiment results show that the proposed method significantly outperforms the state-of-the-art models.

ACR-SA: attention-based deep model through two-channel CNN and Bi-RNN for sentiment analysis

“…We collected this dataset about Afghanistan’s ( ) security status from 29/03/2018 to 21/06/2018. The dataset includes 56 k tweets with positive, negative, and neutral classification ( Kamyab et al, 2018 ; Kamyab, Liu & Adjeisah, 2021 ). However, we consider only positive and negative tweets from this dataset, given that this work explores binary classification problems.…”

“…Currently, the sentiment140 dataset is one of the mostly used and standard datasets for text classification, with 1,048,576 tweets automatically categorized into 248,576 positive and 80,000 negatives. It has been employed in various SOTA studies ( Song et al, 2020 ; Rezaeinia et al, 2019 ; Dos Santos & Gatti, 2014 ; Wang et al, 2016a ; Dang, Moreno-García & De la Prieta, 2020 ; Nguyen & Nguyen, 2019 ; Basiri et al, 2021 ; Alec & Richa Bhayani, 2009 ; Kamyab, Liu & Adjeisah, 2021 ).…”

“…However, neither method achieved satisfactory results for large datasets, such as sentiment140. In our previous study, attention-based CNN and BiLSTM Model for Sentiment Analysis ( Kamyab, Liu & Adjeisah, 2021 ). CNN is employed to learn high-level feature context and, the attention mechanism is applied at the CNN output layers to pay suitable attention to different documents.…”

“…Nevertheless, there is a problem with the input sequence, and the file is eventually decoded to a specific length vector no matter how long the input text sequence is. Our previous work attention-based model utilizes CNN with LSTM (ACL) for sentiment classification ( Kamyab, Liu & Adjeisah, 2021 ). In this model CNN utilizes to extract contextual features and bidirectional LSTM to capture long-term dependencies.…”

Music genres classification by deep learning

“…In addition, recurrent neural networks RNN does not have long-term memory that it can only work on short interval data which limit the training cannot adapt data with long interval or the data with long duration [5], [12]. To conquer the longterm problem in recurrent neural networks RNN, there is a variation of RNN named long short-term memory (LSTM) [20]. This model realizes the capability of learning long-term dependencies by implementing and optimizing the model of RNN.…”