Shrink and Eliminate: A Study of Post-Training Quantization and Repeated Operations Elimination in RNN Models

Abstract

Recurrent neural networks (RNNs) are neural networks (NN) designed for time-series applications. There is a growing interest in running RNNs to support these applications on edge devices. However, RNNs have large memory and computational demands that make them challenging to implement on edge devices. Quantization is used to shrink the size and the computational needs of such models by decreasing weights and activation precision. Further, the delta networks method increases the sparsity in activation vectors by relying on the temporal relationship between successive input sequences to eliminate repeated computations and memory accesses. In this paper, we study the effect of quantization on LSTM-, GRU-, LiGRU-, and SRU-based RNN models for speech recognition on the TIMIT dataset. We show how to apply post-training quantization on these models with a minimal increase in the error by skipping quantization of selected paths. In addition, we show that the quantization of activation vectors in RNNs to integer precision leads to considerable sparsity if the delta networks method is applied. Then, we propose a method for increasing the sparsity in the activation vectors while minimizing the error and maximizing the percentage of eliminated computations. The proposed quantization method managed to compress the four models more than 85%, with an error increase of 0.6, 0, 2.1, and 0.2 percentage points, respectively. By applying the delta networks method to the quantized models, more than 50% of the operations can be eliminated, in most cases with only a minor increase in the error. Comparing the four models to each other under the quantization and delta networks method, we found that compressed LSTM-based models are the most-optimum solutions at low-error-rates constraints. The compressed SRU-based models are the smallest in size, suitable when higher error rates are acceptable, and the compressed LiGRU-based models have the highest number of eliminated operations.

MOHAQ: Multi-Objective Hardware-Aware Quantization of recurrent neural networks

“…Thus, we had a better opportunity to have multiple trials to explore our methodology. In our research, we have shown that SRU and LiGRU-based models can provide smaller compressed models than other RNN models [15]. However, they suffer more from high error rates at high compression ratios.…”

“…In LiGRU, we also apply the integer quantization to the weight matrices only. In our work, we have found that in the case of the LiGRU unit, there exists a weight matrix that is more sensitive to quantization [15]. This matrix is the matrix multiplied by the hiddenstate vector in the candidate state vector computation.…”

“…First, we use post-training quantization as a compression method. Posttraining quantization has become a more and more reliable compression method recently [12][13][14][15]. Thus, the evaluation of one candidate solution would require only running the inference of the NN model.…”

Learning Low-Precision Structured Subnetworks Using Joint Layerwise Channel Pruning and Uniform Quantization

“…For existing solutions, we summarize the results reported in prior work. For solutions that do not apply quantization to the weights or activations, we estimate their precision at 16 bits per element rather than 32 since neural networks can be quantized to 16-bit fixed-point through post-training quantization without significant accuracy degradation [50].…”

Mohaq: Multi-Objective Hardware-Aware Quantization Of Recurrent Neural Networks

“…First, we use post-training quantization as a compression method. Posttraining quantization has become a more and more reliable compression method recently [12][13][14][15]. Thus, the evaluation of one candidate solution would require only running the inference of the NN model.…”