Transform Quantization for CNN Compression

Minimax Optimal Quantization of Linear Models: Information-Theoretic Limits and Efficient Algorithms

“…We now present a lower bound for the asymptotic minimax risk (4) in estimating θ from the linear model (2). The proof sketch in this section is outlined via Lemmas 4.1, 4.2, and 4.3 and build up to our proposed lower bound in Thm.…”

“…Model compression techniques employ lossless and lossy source coding schemes to quantize the model parameters subject to finite precision constraints in order to make them deployable on memory-constrained devices [2,3]. In this work, we adopt an information-theoretic approach to the problem of quantizing linear models without any distributional assumptions on the data or the true model.…”

“…For a learning code Q, the mutual information between the data X (input) and Q(X) (output) is a reasonable proxy for the number of bits B used to design the corresponding encoder-decoder pair (E, D); in other words, the number of bits required for representing the estimate of θ. The marginal distribution of X is determined by W, θ, and v according to (2), and the distribution of the output of the learning code θ ≡ Q(X) is determined solely by the marginal of X and the conditional distribution of θ given X. This idea follows from the information-rate distortion function in rate distortion theory literature [12][Ch.…”

Population Risk Improvement with Model Compression: An Information-Theoretic Approach

“…More recently, an iterative pruning and retraining algorithm to further reduce the size of deep models was proposed in [9,21]. The method of network quantization or weight sharing, i.e., employing a clustering algorithm to group the weights in a neural network, and its variants, including vector quantization [22], soft quantization [23,24], fixed point quantization [25], transform quantization [26], and Hessian weighted quantization [11], have been extensively investigated. Matrix factorization, where low-rank approximation of the weights in neural networks is used instead of the original weight matrix, has also been widely studied in [27][28][29].…”

“…We consider the following two compression algorithms. The first one is the conditional distribution P Ŵ|W in the proof of achievability (26), which requires the knowledge of w * and is denoted as "Oracle". The second one is the well-known K-means clustering algorithm, where the weights in W are grouped into K clusters and represented by the cluster centers in the reconstruction Ŵ.…”