Analog/Mixed-Signal Hardware Error Modeling for Deep Learning Inference

Rekhi, Angad; Zimmer, Brian; Nedovic, Nikola; Liu, Ningxi; Venkatesan, Rangharajan; Wang, Miaorong; Khailany, Brucek; Dally, William J.; Gray, C. Thomas

doi:10.1145/3316781.3317770

Cited by 47 publications

(58 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There exist many different methods of training a neural network with noise that aim to improve the resilience of the model to analog mixed-signal hardware. These include injecting additive noise on the inputs of every layer 20 , on the preactivations 22,23 , or just adding noise on the input data 47 . Moreover, injecting multiplicative Gaussian noise to the weights 34 (σ l δW tr ;ij / jW l ij j) is also defensible regarding the observed noise on the hardware.…”

Section: Discussionmentioning

confidence: 99%

“…As early as in 1994, it was shown that injecting noise on the synaptic weights during training enhances the tolerance to weight perturbations of multi-layer perceptrons, and the application of this technique to analog neural hardware was discussed 34 . Recent works have also proposed to apply noise to the layer inputs or pre-activations in order to improve the network tolerance to hardware noise 20,23 . In this work, we follow the original approach of Murray et al 34 of injecting Gaussian noise to the synaptic weights during training.…”

Section: Resnet Block 1 10 Layersmentioning

confidence: 99%

“…In this way, the model would have to be trained only once and could be deployed on a multitude of different chips. To this end, several works have proposed to inject noise in the training algorithm to the layer inputs 20 , synaptic weights 21 , and pre-activations 22,23 . However, previous demonstrations have generally been limited to rather simple and shallow networks, and experimental validations of the effectiveness of the various approaches have been missing.…”

mentioning

confidence: 99%

“…However, previous demonstrations have generally been limited to rather simple and shallow networks, and experimental validations of the effectiveness of the various approaches have been missing. We are aware of one recent work that analyzed more complex problems such as ImageNet classification 23 , however the hardware model used was rather abstract and no experimental validation was presented.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Accurate deep neural network inference using computational phase-change memory

Joshi¹,

Gallo²,

Haefeli³

et al. 2020

Nat Commun

363

323

View full text Add to dashboard Cite

In-memory computing using resistive memory devices is a promising non-von Neumann approach for making energy-efficient deep learning inference hardware. However, due to device variability and noise, the network needs to be trained in a specific way so that transferring the digitally trained weights to the analog resistive memory devices will not result in significant loss of accuracy. Here, we introduce a methodology to train ResNet-type convolutional neural networks that results in no appreciable accuracy loss when transferring weights to phase-change memory (PCM) devices. We also propose a compensation technique that exploits the batch normalization parameters to improve the accuracy retention over time. We achieve a classification accuracy of 93.7% on CIFAR-10 and a top-1 accuracy of 71.6% on ImageNet benchmarks after mapping the trained weights to PCM. Our hardware results on CIFAR-10 with ResNet-32 demonstrate an accuracy above 93.5% retained over a one-day period, where each of the 361,722 synaptic weights is programmed on just two PCM devices organized in a differential configuration.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Resnet Block 1 10 Layersmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Accurate deep neural network inference using computational phase-change memory

Joshi¹,

Gallo²,

Haefeli³

et al. 2020

Nat Commun

363

323

View full text Add to dashboard Cite

show abstract

“…Deep neural network inference tasks, which are the designated applications for the presented IMC system, can tolerate this small reduction of precision of the MAC operation with usually no loss or in certain cases with insignificant loss in classification accuracy. The effects of ADC quantization, to which any reduced-precision implementation is subjected, are studied in detail in [27].…”

Section: F Noise and Mismatch Impactmentioning

confidence: 99%