Energy Efficient Learning With Low Resolution Stochastic Domain Wall Synapse for Deep Neural Networks

Abstract

We demonstrate extremely low resolution quantized (nominally 5-state) synapses with large stochastic variations in synaptic weight can be energy efficient and achieve reasonably high testing accuracies compared to Deep Neural Networks (DNNs) of similar sizes using floating precision synaptic weights. Specifically, voltage-controlled domain wall (DW) devices demonstrate stochastic behavior and can only encode limited states; however, they are extremely energy efficient during both training and inference. In this study, we propose both in-situ and ex-situ training algorithms, based on modification of the algorithm proposed by Hubara et al. [1] which works well with quantization of synaptic weights, and train several 5layer DNNs on MNIST dataset using 2-, 3-and 5-state DW devices as a synapse. For in-situ training, a separate high precision memory unit preserves and accumulates the weight gradients which prevents the accuracy loss due to weight quantization. For ex-situ training, a precursor DNN is first trained based on weight quantization and characterized DW device model. Moreover, a noise tolerance margin is included in both of the training methods to account for the intrinsic device noise. The highest inference accuracies we obtain after in-situ and ex-situ training are ~ 96.67% and ~96.63% which is very close to the baseline accuracy of ~97.1% obtained from a similar topology DNN having floating precision weights with no stochasticity. Large inter-state interval due to quantized weights and noise tolerance margin enables in-situ training with significantly lower number of programming attempts. Our proposed approach demonstrates a possibility of at least two orders of magnitude energy savings compared to the floating-point approach implemented in CMOS. This approach is specifically attractive for low power intelligent edge devices where the ex-situ learning can be utilized for energy efficient non-adaptive tasks and the in-situ learning can provide the opportunity to adapt and learn in a dynamically evolving environment.

Demonstration of Synaptic Behavior in a Heavy-Metal-Ferromagnetic-Metal-Oxide-Heterostructure-Based Spintronic Device for On-Chip Learning in Crossbar-Array-Based Neural Networks

“…The same thresholding algorithm is used here as in the ideal 5-bit synapse case (black plots in Figure ), but the weight update now deviates from the fixed value of + W or − W as per the nonlinearity and asymmetry of the LTP and LTD curves (Figure (a)). The drop in accuracy for both data sets between the case for ideal 5-bit synapses and that for experimentally obtained 5-bit synapses happens because of the error introduced in the training due to nonlinearity and asymmetry in synaptic behavior. − …”

“…Even though idealistic simulations show extremely linear and symmetric domain-wall and skyrmion motion, it is hard to achieve such motion experimentally given the presence of defects, thermal noise, and other nonidealities. But most studies on spintronic synaptic devices, reported thus far, are based on idealistic micromagnetic simulations of the devices (leading to very linear and symmetric LTP and LTD) and subsequent device-system co-design and co-simulations of crossbar arrays based on these devices. ,,,− Unless the synaptic device model is based on actual experiments, it does not capture nonlinearities in LTP and LTD, reduction in bit resolution, cycle-to-cycle variations, etc., and this leads to the underestimation of errors in classification made by crossbar arrays simulated based on it. − …”

Antiferromagnetic skyrmion based shape-configured leaky-integrate-fire neuron device

“…Here, ɳ is defined as a random variable having the Gaussian distribution with zero mean and unit variance and independent in each of the three Cartesian coordinates obtained at each time step. kB, T, V, and ∆𝑡 are the Boltzmann constant, temperature, volume of each simulation cell, and time step, respectively [44,45]. The dimension of the nanotrack is considered as 324x120x2 nm 3 for the simulations.…”

Magnetic skyrmions and domain walls for logical and neuromorphic computing

“…Due to the tunability of DW-MTJ neurons and synapses, there are opportunities to design neural network accelerators that can benefit from the characteristics outlined in the previous sections. Due to the linear and symmetric updates that have been demonstrated for DW-MTJ devices [126,127], the devices are effective artificial synapses for deep neural networks. On top of this, the tunability of the magnetic dynamics of the DW-MTJ synapses can be leveraged for specific types of deep neural networks.…”