Neuron-Level Fuzzy Memoization in RNNs

Abstract

Recurrent Neural Networks (RNNs) are a key technology for applications such as automatic speech recognition or machine translation. Unlike conventional feed-forward DNNs, RNNs remember past information to improve the accuracy of future predictions and, therefore, they are very effective for sequence processing problems. For each application run, each recurrent layer is executed many times for processing a potentially large sequence of inputs (words, images, audio frames, etc.). In this paper, we make the observation that the output of a neuron exhibits small changes in consecutive invocations. We exploit this property to build a neuron-level fuzzy memoization scheme, which dynamically caches the output of each neuron and reuses it whenever it is predicted that the current output will be similar to a previously computed result, avoiding in this way the output computations. The main challenge in this scheme is determining whether the new neuron's output for the current input in the sequence will be similar to a recently computed result. To this end, we extend the recurrent layer with a much simpler Bitwise Neural Network (BNN), and show that the BNN and RNN outputs are highly correlated: if two BNN outputs are very similar, the corresponding outputs in the original RNN layer are likely to exhibit negligible changes. The BNN provides a low-cost and effective mechanism for deciding when fuzzy memoization can be applied with a small impact on accuracy. We evaluate our memoization scheme on top of a state-of-the-art accelerator for RNNs, for a variety of different neural networks from multiple application domains. We show that our technique avoids more than 24.2% of computations, resulting in 18.5% energy savings and 1.35x speedup on average. CCS CONCEPTS • Computer systems organization → Neural Networks; • Computing Methodologies → Machine Learning;

Mixture-of-Rookies: Saving DNN Computations by Predicting ReLU Outputs

“…Even though most neurons exhibit a high correlation, a significant number of neurons have moderate or even low correlations. This observation is consistent with the observations made by Anderson et al [18] and more recently, Silfa et al [9]. A predictor based on 1-bit weights for a neuron with a low self-correlation coefficient is expected to make frequent mistakes, and consequently, reduce the overall accuracy of the DNN.…”

“…First, the dot-product between 1-bit valued vectors does not require multipliers, simplifying the hardware by a large extent. Second, since the 1-bit weights are obtained from the sign bits of the full precision weights [9], they do not incur in any memory footprint overhead since they do not have to be stored separately, but can be obtained directly from the full precision weights.…”

“…The first component is based on the observation that the output of the binarized dot product, i.e. after converting inputs and weights to 1-bit, shows high linear correlation with the original 8-bit dot product [9]. For this purpose, we use a subset of the training set to perform a linear regression between binarized and 8-bit outputs, obtaining a fitted line for each neuron/filter.…”

Md-Hm

“…The memoization technique has been studied in many fields. Franyell et al [71] propose a fuzzy memoization scheme that avoids more than 24.2% of computation for RNN training. Liu et al [44] replace a long sequence of instructions with a two-level lookup table.…”

Boosting LSTM Performance Through Dynamic Precision Selection

“…DeltaRNN [20] and the work in [21] exploit temporal coherency of the LSTM data to reuse computations and avoid redundant memory accesses. Work in [22] improves RNN energy efficiency by skipping computations, since previous calculations are memoized and reused in future evaluations. We focus on changing the bit-width at runtime, and computations are never skipped.…”