Simple Recurrent Units for Highly Parallelizable Recurrence

Abstract

Common recurrent neural architectures scale poorly due to the intrinsic difficulty in parallelizing their state computations. In this work, we propose the Simple Recurrent Unit (SRU), a light recurrent unit that balances model capacity and scalability. SRU is designed to provide expressive recurrence, enable highly parallelized implementation, and comes with careful initialization to facilitate training of deep models. We demonstrate the effectiveness of SRU on multiple NLP tasks. SRU achieves 5-9x speed-up over cuDNN-optimized LSTM on classification and question answering datasets, and delivers stronger results than LSTM and convolutional models. We also obtain an average of 0.7 BLEU improvement over the Transformer model (Vaswani et al., 2017) on translation by incorporating SRU into the architecture. 1

Metric Learning for Dynamic Text Classification

“…We apply the same encoder architecture throughout all experiments. We use a 4 layer recurrent neural network, with SRU cells (Lei et al, 2018) and a hidden size of 128. We use pretrained GloVe embeddings (Pennington et al, 2014), which are fixed during training.…”

Building a Production Model for Retrieval-Based Chatbots

“…While many of the models cited above implement their RNNs with an LSTM (Hochreiter and Schmidhuber, 1997), we instead use an SRU (Lei et al, 2018). SRU uses light recurrence, which makes it highly parallelizable, and Lei et al (2018) showed that it trains 5-9x faster than cuDNN LSTM. SRU also exhibits a significant speedup in inference time compared to LSTM (by a factor of 4.1x in our experiments), which is particularly relevant in a production setting.…”

“…SRU also exhibits a significant speedup in inference time compared to LSTM (by a factor of 4.1x in our experiments), which is particularly relevant in a production setting. Furthermore, Lei et al (2018) showed that SRU matches or exceeds the performance of models using LSTMs or the Transformer architecture (Vaswani et al, 2017) on a number of NLP tasks, meaning significant speed gains can be achieved without a drop in performance.…”

“…We present a dual encoder architecture that is optimized to select among as many as 10,000 responses within a couple tens of milliseconds. The model makes use of a fast recurrent network implementation (Lei et al, 2018) and multiheaded attention and achieves over a 4.1x inference speedup over traditional encoders such as LSTM (Hochreiter and Schmidhuber, 1997). The independent dual encoding allows pre-computing the embeddings of candidate responses, thereby making the approach highly scalable with the size of the whitelist.…”

Flambé: A Customizable Framework for Machine Learning Experiments

“…In this experiment, we fine-tune the BERT model on two standard text classification benchmarks: TREC (Li and Roth, 2002) and Sentiment Treebank (Socher et al, 2013). We then apply knowledge distillation to reduce the BERT model to a simple 4 layer, 256 units, SRU network (Lei et al, 2018). This is a typical multistage experiment with preprossessing, fine tuning, and distillation stages.…”