A BERT-Based Hybrid Short Text Classification Model Incorporating CNN and Attention-Based BiGRU

Abstract

Short text classification is a research focus for natural language processing (NLP), which is widely used in news classification, sentiment analysis, mail filtering and other fields. In recent years, deep learning techniques are applied to text classification and has made some progress. Different from ordinary text classification, short text has the problem of less vocabulary and feature sparsity, which raise higher request for text semantic feature representation. To address this issue, this paper propose a feature fusion framework based on the Bidirectional Encoder Representations from Transformers (BERT). In this hybrid method, BERT is used to train word vector representation. Convolutional neural network (CNN) capture static features. As a supplement, a bi-gated recurrent neural network (BiGRU) is adopted to capture contextual features. Furthermore, an attention mechanism is introduced to assign the weight of salient words. The experimental results confirmed that the proposed model significantly outperforms the other state-of-the-art baseline methods.

WITHDRAWN: Emotion Detection in Arabic Short Text: A Deep-Learning Approach

“…An improved version of the RNN that combines both forward and backward GRU is called the BiGRU model. As the final result, it successfully mixes the data that was taken from the two directions of the sequence 29 . In contrast to LSTM networks, bidirectional GRUs do not have an output gate.…”

“…The next convolutional layer uses different-sized convolution kernels to perform convolution operations on the word vector matrix in order to extract pertinent local features. Reducing the dimensionality of the data and reducing the likelihood of overfitting are the main goals of the pooling layer 29 . Figure 10 shows the architectural layout of the CNN.…”

“…improvements 29 . The Bi-GRU model, which accurately classifies emotions by incorporating the Bi-GRU network and other mechanisms, is the classification technique used in this study 34 .…”

Research on text classification technology based on natural language processing

“…The principle of classification is to summarize a classification function or build a classification model based on the existing data, which is what we usually call a classifier. Through the classification function or classifier, the data in the database can be classified into a specific category according to its characteristics, and finally applied to the prediction of the data [8]. The text classification algorithm flow is shown in Figure 2.…”

Sentiment analysis of Indonesian datasets based on a hybrid deep-learning strategy

“…In this paper, our work determines the sentiment analysis in a short document or sentences. The challenges faced in the classification of short documents are their scant and irregular structures [ 6 ]. Various techniques in sentiment classification, including traditional methods, machine learning, and hybrid models, have been proposed by researchers to address several challenges.…”