Training Neural Nets using only an Approximate Tableless LNS ALU

Arnold, Mark G.; Chester, Ed; Johnson, Corey

doi:10.1109/asap49362.2020.00020

Cited by 7 publications

(7 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, it can be seen in Table 2 that for very short word lengths, the average arithmetic error for bases other than base-2 is much lower than 0.25 × ULP LNS , especially for the subtraction table. For example, base-1.417 gives an arithmetic error of 17.1544% of ULP LNS for Q(2, 2) while base 1.415 gives 19.9971% of ULP LNS (in the log domain) for Q (2,3). These errors converge to around 0.25 × ULP LNS as we increase the word length.…”

Section: Evaluation Of Arithmetic Errormentioning

confidence: 97%

“…Vogel et al [42], present a low word length-based quantization method for LNS to be used in a neural network and have shown LNS to achieve 22.3% lower power as compared to an 8-bit fixed-point-based design. Arnold et al [3] present an approach to implement back propagation using tableless LNS ALU based on modified Mitchell's method [28] and achieve one third reduction in the hardware resources as compared to a conventional fixed-point implementation.…”

Section: Related Workmentioning

confidence: 99%

“…For example, for the case of Q(2, 2), the input word length is 5-bits while the output word length is varied from 5 to 16 bits. The results for two different total input word length, Q(2, 2) and Q (3,3), are shown in Figure 8, where each value represents the minimum percentage average absolute arithmetic error. For each value, the base that gives this minimum value is unique.…”

Section: Higher-precision Tables For Very Short Word Lengthsmentioning

confidence: 99%

See 2 more Smart Citations

Low-precision Logarithmic Number Systems

Alam

Garland

Gregg

2021

ACM Trans. Archit. Code Optim.

View full text Add to dashboard Cite

Logarithmic number systems (LNS) are used to represent real numbers in many applications using a constant base raised to a fixed-point exponent making its distribution exponential. This greatly simplifies hardware multiply, divide, and square root. LNS with base-2 is most common, but in this article, we show that for low-precision LNS the choice of base has a significant impact. We make four main contributions. First, LNS is not closed under addition and subtraction, so the result is approximate. We show that choosing a suitable base can manipulate the distribution to reduce the average error. Second, we show that low-precision LNS addition and subtraction can be implemented efficiently in logic rather than commonly used ROM lookup tables, the complexity of which can be reduced by an appropriate choice of base. A similar effect is shown where the result of arithmetic has greater precision than the input. Third, where input data from external sources is not expected to be in LNS, we can reduce the conversion error by selecting a LNS base to match the expected distribution of the input. Thus, there is no one base that gives the global optimum, and base selection is a trade-off between different factors. Fourth, we show that circuits realized in LNS require lower area and power consumption for short word lengths.

show abstract

Section: Evaluation Of Arithmetic Errormentioning

confidence: 97%

Section: Related Workmentioning

confidence: 99%

Section: Higher-precision Tables For Very Short Word Lengthsmentioning

confidence: 99%

See 1 more Smart Citation

Low-precision Logarithmic Number Systems

Alam

Garland

Gregg

2021

ACM Trans. Archit. Code Optim.

View full text Add to dashboard Cite

show abstract

“…This has prevented its adoption as a mainstream format for general-purpose computing. A common option is to use approximate LNS adder/subtracters [17,18,19,20]. The alternative approach proposed in this work is to use very low precisions (less than 8 bits) for which accurate LNS arithmetic is cheap.…”

Section: A Logarithmic Number Systems (Lns)mentioning

confidence: 99%

Low-precision logarithmic arithmetic for neural network accelerators

Maxime

Dinechin

Pétrot

2022

2022 IEEE 33rd International Conference on Application-Specific Systems, Architectures and Processors (ASAP)

View full text Add to dashboard Cite

Resource requirements for hardware acceleration of neural networks inference is notoriously high, both in terms of computation and storage. One way to mitigate this issue is to quantize parameters and activations. This is usually done by scaling and centering the distributions of weights and activations, on a kernel per kernel basis, so that a low-precision binary integer representation can be used. This work studies low-precision logarithmic number system (LNS) as an efficient alternative.Firstly, LNS has more dynamic than fixed-point for the same number of bits. Thus, when quantizing MNIST and CIFAR reference networks without retraining, the smallest format size achieving top-1 accuracy comparable to floating-point is 1 to 3 bits smaller with LNS than with fixed-point. In addition, it is shown that the zero bit of classical LNS is not needed in this context, and that the sign bit can be saved for activations.Secondly, low-precision LNS enables efficient inference architectures where 1/ multiplications reduce to additions; 2/ the weighted inputs are converted to classical linear domain, but the tables needed for this conversion remain very small thanks to the low precision; and 3/ the conversion of the output activation back to LNS can be merged with an arbitrary activation function.The proposed LNS neuron is detailed and its implementation on FPGA is shown to be smaller and faster than a fixed-point one for comparable accuracy.

show abstract

“…Vogel et al [26], present a low word length based quantization method for LNS to be used in a neural network and have shown LNS to achieve 22.3% lower power as compared to an 8-bit fixed point based design. Arnold et al [27] present an approach to implement back propagation using tableless LNS ALU based on modified Mitchell's method [28] and achieve one third reduction in the hardware resources as compared to a conventional fixed-point implementation.…”

Section: Related Workmentioning

confidence: 99%

Low precision logarithmic number systems: Beyond base-2

Alam,

Garland,

Gregg

2021

Preprint

View full text Add to dashboard Cite

Logarithmic number systems (LNS) are used to represent real numbers in many applications using a constant base raised to a fixed-point exponent making its distribution exponential. This greatly simplifies hardware multiply, divide and square root. LNS with base-2 is most common, but in this paper we show that for low-precision LNS the choice of base has a significant impact. We make four main contributions. First, LNS is not closed under addition and subtraction, so the result is approximate. We show that choosing a suitable base can manipulate the distribution to reduce the average error. Second, we show that low-precision LNS addition and subtraction can be implemented efficiently in logic rather than commonly used ROM lookup tables, the complexity of which can be reduced by an appropriate choice of base. A similar effect is shown where the result of arithmetic has greater precision than the input. Third, where input data from external sources is not expected to be in LNS, we can reduce the conversion error by selecting a LNS base to match the expected distribution of the input. Thus, there is no one base which gives the global optimum, and base selection is a trade-off between different factors. Fourth, we show that circuits realized in LNS require lower area and power consumption for short word lengths.

show abstract

Training Neural Nets using only an Approximate Tableless LNS ALU

Cited by 7 publications

References 12 publications

Low-precision Logarithmic Number Systems

Low-precision Logarithmic Number Systems

Low-precision logarithmic arithmetic for neural network accelerators

Low precision logarithmic number systems: Beyond base-2

Contact Info

Product

Resources

About