Weight discretization paradigm for optical neural networks

Fiesler, Emile; Choudry, A.; Caulfield, H. John

doi:10.1117/12.20700

Cited by 68 publications

(35 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that pre-training with continuous weights is not necessary, but is often included for faster convergence. The discretization of the weights described by Fiesler et al 16] uses d discretization levels with d= 2 , 3 , 5 , 7 , o r 9 . T h e discretization levels are symmetric around zero, except ford = 2 and are equidistant:…”

Section: Discussionmentioning

confidence: 99%

Discrete all-positive multilayer perceptrons for optical implementation

1998

Self Cite

View full text Add to dashboard Cite

Abstract. All-optical multilayer perceptrons di er in various ways from the ideal neural network model. Examples are the use of non-ideal activation functions which are truncated, asymmetric, and have a non-standard gain, restriction of the network parameters to non-negative v alues, and the limited accuracy of the weights. In this paper, a backpropagation-based learning rule is presented that compensates for these non-idealities and enables the implementation of all-optical multilayer perceptrons where learning occurs under control of a computer. The good performance of this learning rule, even when using a small number of weight l e v els, is illustrated by a series of computer simulations incorporating the non-idealities. IDIAP{RR 97-02

show abstract

Section: Discussionmentioning

confidence: 99%

Discrete all-positive multilayer perceptrons for optical implementation

1998

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, direct quantization of the trained floating-point weights does not yield good results. Therefore, we employ the weight quantization strategy similar to the algorithms proposed in [7,13] to retrain the weights after the direct quantization. Also, internal signals (output values of the units) are uniformly quantized from 0 to 1.…”

Section: Fixed-point Dnn Design For Phoneme Recognitionmentioning

confidence: 99%

X1000 real-time phoneme recognition VLSI using feed-forward deep neural networks

Kim

Hwang

Sung

2014

2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Deep neural networks show very good performance in phoneme and speech recognition applications when compared to previously used GMM (Gaussian Mixture Model)-based ones. However, efficient implementation of deep neural networks is difficult because the network size needs to be very large when high recognition accuracy is demanded. In this work, we develop a digital VLSI for phoneme recognition using deep neural networks and assess the design in terms of throughput, chip size, and power consumption. The developed VLSI employs a fixed-point optimization method that only uses +Δ, 0, and -Δ for representing each of the weight. The design employs 1,024 simple processing units in each layer, which however can be scaled easily according to the needed throughput, and the throughput of the architecture varies from 62.5 to 1,000 times of the real-time processing speed.

show abstract

“…In order to solve this problem, methods of reducing the size of a CNN model or designing an efficient CNN structure have emerged as core fields in recent deep neural network research. Representative methods include pruning [9]- [12], quantization [13]- [15], knowledge distillation [16]- [18], weight sharing [19]- [22], and efficient structural design methods, e.g., Depthwise Separable Convolution [23]- [26]. These methods are widely used to compress the size of the CNN model.…”

Section: Introductionmentioning

confidence: 99%

Compressing Neural Networks With Inter Prediction and Linear Transformation

Lee

Bae

2021

IEEE Access

View full text Add to dashboard Cite

Because of resource-constrained environments, network compression has become an essential part of deep neural networks research. In this paper, we found a mutual relationship between kernel weights termed as Inter-Layer Kernel Correlation (ILKC). The kernel weights between two different convolution layers share a substantial similarity in shapes and values. Based on this relationship, we propose a new compression method, Inter-Layer Kernel Prediction (ILKP), which represents convolutional kernels with fewer bits through similarity between kernel weights in convolutional neural networks. Furthermore, to effectively adapt the inter prediction scheme from video coding technology, we integrate a linear transformation into the prediction scheme, which significantly enhances compression efficiency. The proposed method achieved 93.77% top-1 accuracy with 4.1× compression ratio compared to the ResNet110 baseline model on CIFAR10. It means that 0.04% top-1 accuracy improvement was achieved by using less memory footprint. Moreover, incorporating quantization, the proposed method achieved a 13× compression ratio with little performance degradation compared to the ResNets baseline model trained on CIFAR10 and CIFAR100.

show abstract

Weight discretization paradigm for optical neural networks

Cited by 68 publications

References 0 publications

Discrete all-positive multilayer perceptrons for optical implementation

Discrete all-positive multilayer perceptrons for optical implementation

X1000 real-time phoneme recognition VLSI using feed-forward deep neural networks

Compressing Neural Networks With Inter Prediction and Linear Transformation

Contact Info

Product

Resources

About