Noise Injection Adaption

Abstract

In this work, we investigate various non-ideal effects (Stuck-At-Fault (SAF), IR-drop, thermal noise, shot noise, and random telegraph noise) of ReRAM crossbar when employing it as a dot-product engine for deep neural network (DNN) acceleration. In order to examine the impacts of those non-ideal effects, we first develop a comprehensive framework called PytorX based on mainstream DNN pytorch framework. PytorX could perform end-to-end training, mapping, and evaluation for crossbar-based neural network accelerator, considering all above discussed non-ideal effects of ReRAM crossbar together. Experiments based on PytorX show that directly mapping the trained large scale DNN into crossbar without considering these non-ideal effects could lead to a complete system malfunction (i.e., equal to random guess) when the neural network goes deeper and wider. In particular, to address SAF side effects, we propose a digital SAF error correction algorithm to compensate for crossbar output errors, which only needs one-time profiling to achieve almost no system accuracy degradation. Then, to overcome IR drop effects, we propose a Noise Injection Adaption (NIA) methodology by incorporating statistics of current shift caused by IR drop in each crossbar as stochastic noise to DNN training algorithm, which could efficiently regularize DNN model to make it intrinsically adaptive to non-ideal ReRAM crossbar. It is a one-time training method without the request of retraining for every specific crossbar. Optimizing system operating frequency could easily take care of rest non-ideal effects. Various experiments on different DNNs using image recognition application are conducted to show the efficacy of our proposed methodology.

Current injection: a hardware method of adapting non-ideal effects of ReRAM based deep learning accelerator

“…Note that such current injection is added to the initial DNN training for crossbar mapping, and no re-training is needed for each particular accelerator. • We validate our result using PytorX [8] on MNIST, CIFAR-10, and Imagenet dataset, respectively.…”

“…Compared to the circuit level model proposed in [22], our model is independent of technology and can be applied to other memristor devices with the Gaussian set process. Therefore our model is compatible with the behavioral level simulation method used for large scale networks [8,23]. The only necessary trimming of our model is to change , , , and based on device characterization.…”

“…However, the process needs to be repeated whenever the crossbar has different input signals. In [8], He et al employ the statistics of current drift introduced by IR drop as a stochastic noise source in the training phase of DNN. However, to obtain the accurate dropout data, the nodal analysis was utilized, imposing a huge computation burden on the accelerator.…”

“…On the other hand, due to analog computing's nature, the accuracy of the ReRAM crossbar's output is hampered by many non-ideal effects, such as wire resistance (IR drop), process variation, and electrical noise. [8]. Many recent works have investigated such issues, which are summarized as follows.…”

“…Moreover, typical anti-nonideal effect retraining considers and addresses the deterministic non-ideal effect, leaving non-ideal effects with stochastic behavior remain unsolved. In [8], He et al have proven that by injecting noise into the network during the training phase, the crossbar's non-ideal effects can be adapted. However, their method is based on obtaining the noise profile of each crossbar during training, which is a huge computation burden.…”