Multilayer feedforward neural networks with single powers-of-two weights

Tang, C.Z.; Kwan, Hon Keung

doi:10.1109/78.229903

Cited by 57 publications

(29 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has been concluded in [12] that the weight distribution in a neural network resembles a normal distribution when the word width is long enough. If the fan-in number is large enough (N w > 30), then according to the central limit theorem, the weighted sum can be seen as having N w independent random input variables, and the probability density function of their weighted sum can then be considered to be a normal distribution.…”

Section: The Fixed-point Quantization Analysis Of Output Functionmentioning

confidence: 99%

Analysis of quantization effects on high-order function neural networks

Jiang

Gielen

2007

Appl Intell

View full text Add to dashboard Cite

In this paper we investigate the combined effects of quantization and clipping on high-order function neural networks (HOFNN). Statistical models are used to analyze the effects of quantization in a digital implementation. We analyze the performance degradation caused as a function of the number of fixed-point and floating-point quantization bits under the assumption of different probability distributions for the quantized variables, and then compare the training performance between situations with and without weight clipping. We establish and analyze the relationships for a true nonlinear neuron between inputs and outputs bit resolution, training and quantization methods, network order and performance degradation, all based on statistical models, and for on-chip and off-chip training. Our experimental simulation results verify the presented theoretical analysis.

show abstract

Section: The Fixed-point Quantization Analysis Of Output Functionmentioning

confidence: 99%

Analysis of quantization effects on high-order function neural networks

Jiang

Gielen

2007

Appl Intell

View full text Add to dashboard Cite

show abstract

“…However, direct quantization of the trained floating-point weights does not yield good results. Therefore, we employ the weight quantization strategy similar to the algorithms proposed in [7,13] to retrain the weights after the direct quantization. Also, internal signals (output values of the units) are uniformly quantized from 0 to 1.…”

Section: Fixed-point Dnn Design For Phoneme Recognitionmentioning

confidence: 99%

“…There are several, such as analog, digital, hybrid, and FPGA implementation approaches for neural networks [5,6,7,8,9,10,11]. However, the number of neurons in a layer is restricted to a small number in those implementations.…”

Section: Introductionmentioning

confidence: 98%

X1000 real-time phoneme recognition VLSI using feed-forward deep neural networks

Kim

Hwang

Sung

2014

2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Deep neural networks show very good performance in phoneme and speech recognition applications when compared to previously used GMM (Gaussian Mixture Model)-based ones. However, efficient implementation of deep neural networks is difficult because the network size needs to be very large when high recognition accuracy is demanded. In this work, we develop a digital VLSI for phoneme recognition using deep neural networks and assess the design in terms of throughput, chip size, and power consumption. The developed VLSI employs a fixed-point optimization method that only uses +Δ, 0, and -Δ for representing each of the weight. The design employs 1,024 simple processing units in each layer, which however can be scaled easily according to the needed throughput, and the throughput of the architecture varies from 62.5 to 1,000 times of the real-time processing speed.

show abstract

“…2. We don't perform piecewise quantization on weights as [9], because we find DNN is more susceptible to the quantization error of weights than to that of activations, which is shown in Fig. 2.…”

Section: Quantizing Activations Into Powers-of-two Integersmentioning

confidence: 99%

Speeding up Deep Neural Networks in Speech Recognition with Piecewise Quantized Sigmoidal Activation Function

Xing

Zhao

Yan

2016

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYThis paper proposes a new quantization framework on activation function of deep neural networks (DNN). We implement fixedpoint DNN by quantizing the activations into powers-of-two integers. The costly multiplication operations in using DNN can be replaced with lowcost bit-shifts to massively save computations. Thus, applying DNN-based speech recognition on embedded systems becomes much easier. Experiments show that the proposed method leads to no performance degradation.

show abstract

Multilayer feedforward neural networks with single powers-of-two weights

Cited by 57 publications

References 7 publications

Analysis of quantization effects on high-order function neural networks

Analysis of quantization effects on high-order function neural networks

X1000 real-time phoneme recognition VLSI using feed-forward deep neural networks

Speeding up Deep Neural Networks in Speech Recognition with Piecewise Quantized Sigmoidal Activation Function

Contact Info

Product

Resources

About