Linear Computation Coding

Abstract

We introduce the new concept of computation coding. For linear functions, we present an algorithm to reduce the computational cost of multiplying an arbitrary given matrix with an unknown vector. It decomposes the given matrix into the product of codebook and wiring matrices whose entries are either zero or signed integer powers of two.For a typical implementation of deep neural networks, the proposed algorithm reduces the number of required addition units several times. To achieve the accuracy of 16-bit signed integer arithmetic for 4k-vectors, no multipliers and only 1.5 adders per matrix entry are needed.

Linear Computation Coding: A Framework for Joint Quantization and Computing

“…While for large matrices the performance is hardly affected, for small matrices this does make a difference. This is one of several reasons why, in the preliminary conference versions of this work [4,42,43], the decomposition algorithm does not perform so well for small matrices.…”

“…in terms of the zero norm. The approximation (20) was used in the preliminary conference versions of this work [4,42,43]. In the sequel, we continue with the exact number of additions as given in (16).…”

“…We prefer to show the median error over the average error that was used in the conference versions [4,43] of this work. The average error is strongly influenced by rare events, i.e., codebook matrices with many close to collinear columns.…”

“…This does not imply, however, that small matrix size is beneficial. Contrarily, the number of additions per matrix entry is given in (43), which clearly decays with increasing matrix size.…”

“…For an IID Gaussian target matrix with K = 256 columns, we compare various codebook matrices in Figure 5. In contrast to Figure 1 in [43], we use a single wiring matrix to avoid the effects of codebook evolution. The wiring matrix is designed as described in Section 4.2.2.…”