Attention Pooling-Based Bidirectional Gated Recurrent Units Model for Sentimental Classification

Abstract

Recurrent neural network (RNN) is one of the most popular architectures for addressing variable sequence text, and it shows outstanding results in many natural language processing (NLP) tasks and remarkable performance in capturing long-term dependencies. Many models have achieved excellent results based on RNN. However, most of these models overlook the locations of the keywords in a sentence and the semantic connections in different directions. As a consequence, these methods do not make full use of the available information. Considering that different words in a sequence usually have different importance, in this paper, we propose bidirectional gated recurrent units (BGRUs) which integrates a novel attention pooling mechanism with max-pooling operation to force the model to pay attention to the keywords in a sentence and maintain the most meaningful information of the text automatically. The presented model allows to encode longer sequences. Thus, it not only prevents important information from being discarded but also can be used to filter noises. To avoid full exposure of content without any control, we add an output gate to the GRU, which is named as text unit. The proposed model was evaluated on multiple tasks, including sentimental classification, movie review data, and a subjective classification dataset. Experimental results show that our model can achieve excellent performance on these tasks.

Sentiment Analysis of Review Text Based on BiGRU-Attention and Hybrid CNN

“…The self-attention mechanism performs parallel computation on text, thus reducing storage space and speeding up operation. Zhang et al [9] propose the BGRU-Att-pooling model, which applies an element-wise multiplication attention mechanism to analyze the semantic features of the text, and utilizes 2D maximum pooling to retain the most important feature. The optimized effect of reducing the complexity of the model is fulfilled by reducing redundant features.…”

“…GRU changed the calculation method of the hidden layer of RNN, and it adjusted the three gates in LSTM into reset gates and update gates, and combined candidate hidden states to control the flow of text information at different times [9], making the model easily extract information from previous texts. GRU was calculated according to the following formulae.…”

“…For example, Apple or Xiaomi mean differently under different contexts, which refer to the mobile phone or the food. RNN can comprehensively consider the global information of the text, but the long-term looping leads to insufficient attention to key words [9]. The attention mechanism can distinguish the importance of different words, but for very long text sequences, a large storage space is required to save the similarity between words [10].…”

Intelligent Case Assignment Method Based on the Chain of Criminal Behavior Elements

“…The process can be formalized as: behave = TransformerEncoder(ACT) (6) Attention Layer The attention mechanism [33][34][35] lets the model capture the whole traffic dynamics in the input sequence. Inspired by [36,37], we use the attention mechanism as an attention pooling mechanism, which can pay attention to the key element in the chain of criminal behavior elements and maintains the most meaningful information of the facts of the case. The formula is defined as follows:…”

Weight asynchronous update: Improving the diversity of filters in a deep convolutional network

“…In the past few years, deep learning methods based on convolutional neural networks (CNNs) have obtained significant achievements in machine vision [1,2], shape representation [3][4][5], speech recognition [6,7], natural language processing [8][9][10], etc. In particular, many advanced deep convolutional networks have been proposed to handle visual tasks.…”