Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

Abstract

Recurrent neural networks (RNNs) are known to be difficult to train due to the gradient vanishing and exploding problems and thus difficult to learn long-term patterns and construct deep networks. To address these problems, this paper proposes a new type of RNNs with the recurrent connection formulated as Hadamard product, referred to as independently recurrent neural network (IndRNN), where neurons in the same layer are independent of each other and connected across layers. The gradient vanishing and exploding problems are solved in IndRNN, and thus long-term dependencies can be learned. Moreover, an IndRNN can work with non-saturated activation functions such as ReLU (rectified linear unit) and be still trained robustly. Different deeper IndRNN architectures, including the basic stacked IndRNN, residual IndRNN and densely connected IndRNN, have been investigated, all of which can be much deeper than the existing RNNs. Furthermore, IndRNN reduces the computation at each time step and can be over 10 times faster than the commonly used Long short-term memory (LSTM). Experimental results have shown that the proposed IndRNN is able to process very long sequences and construct very deep networks. Better performances have been achieved on various tasks with IndRNNs compared with the traditional RNN and LSTM.unit (GRU) [15] have been proposed to address the gradient problems. However, the use of the hyperbolic tangent and the sigmoid functions as the activation function in these variants results in gradient decay over layers. Consequently, construction and training of a deep LSTM or GRU based RNN network is practically difficult.On the other hand, the existing RNN models share the same component σ(Wx t + Uh t−1 + b) in (1), where the recurrent connection connects all the neurons through time. This makes it hard to interpret and understand the roles of each individual neuron (e.g., what patterns each neuron responds to) without considering the others. Moreover, with the recurrent connections, matrix product is performed at each time step and the computation cannot be easily paralleled, leading to a very time-consuming process when dealing with long sequences.In this paper, we propose a new type of RNN, referred to as independently recurrent neural network (IndRNN). In the proposed IndRNN, the recurrent inputs are processed with the Hadamard product as h t = σ(Wx t +u h t−1 +b). This provides a number of advantages over the traditional RNNs including:• Able to process longer sequences: the gradient vanishing and exploding problem is solved by regulating the recurrent weights, and long-term memory can be kept in order to process long sequences. Experiments have demonstrated that an IndRNN can well process sequences over 5000 steps. • Able to construct deeper networks: multiple layers of IndRNNs can be efficiently stacked, especially with skip-connection and dense connection, to increase the depth of the network. An example of 21-layer residual IndRNN and deep densely connected In-dRNN are demonstrated in the experiments. ...

Deep unsupervised learning of turbulence for inflow generation at various Reynolds numbers

“…We aim to obtain z t+1 , which can generate the flow information at the next time step, by advancing z t in a low dimension. Here, IndyLSTM [34,35], a kind of recurrent neural networks (RNNs), is used to update z t to z t+1 . A similar approach that applies RNNs to the time advancement in a low dimension was used for the prediction of unsteady flows [36,37,14].…”

DeepAffinity: Interpretable Deep Learning of Compound-Protein Affinity through Unified Recurrent and Convolutional Neural Networks

“…For protein representation, we have chosen SSE as the resolution for interpretability due to the known sequence-size limitation of RNN models (Li et al, 2018). One can easily increase the resolution to residuelevel by simply feeding to our models amino-acid sequences (preferentially of length below 1,000) instead of SPS sequences, but needs to be aware of the much increased computational burden and much worse convergence when training RNNs.…”

“…All these are achieved with a much smaller alphabet of size 76 , which leads to around 100-times more compact representation of a protein sequence than the baseline. In addition, the SPS sequences are much shorter than aminoacid sequences and prevents convergence issues when training RNN and LSTM for sequences longer than 1,000 (Li et al, 2018).…”

A Comparison of Pooling Methods on LSTM Models for Rare Acoustic Event Classification

“…However, this naïve method may suffer from long-term memory loss. Li et al [27] showed that LSTM models could only memorize less than 1,000 steps. Our experiments shown in Sec.…”

“…4) were calculated on 1.7M mixtures synthesized with the setup described in Sec.3.3. Memory retaining Li et al [27] showed that LSTM can only keep a mid-range memory (about 500-1,000 time steps). To check if LSTM models have a similar memory forgetting issue on AEC, we can look at the red curves of 'LastFrame' in Fig.…”