High-Accuracy Inference in Neuromorphic Circuits using Hardware-Aware Training

Abstract

Neuromorphic Multiply-And-Accumulate (MAC) circuits utilizing synaptic weight elements based on SRAM or novel Non-Volatile Memories (NVMs) provide a promising approach for highly efficient hardware representations of neural networks. NVM density and robustness requirements suggest that off-line training is the right choice for "edge" devices, since the requirements for synapse precision are much less stringent. However, off-line training using ideal mathematical weights and activations can result in significant loss of inference accuracy when applied to non-ideal hardware. Non-idealities such as multi-bit quantization of weights and activations, nonlinearity of weights, finite max/min ratios of NVM elements, and asymmetry of positive and negative weight components all result in degraded inference accuracy. In this work, it is demonstrated that non-ideal Multi-Layer Perceptron (MLP) architectures using low bitwidth weights and activations can be trained with negligible loss of inference accuracy relative to their Floating Point-trained counterparts using a proposed off-line, continuously differentiable HW-aware training algorithm. The proposed algorithm is applicable to a wide range of hardware models, and uses only standard neural network training methods. The algorithm is demonstrated on the MNIST and EMNIST datasets, using standard MLPs.

Improving Noise Tolerance of Mixed-Signal Neural Networks

“…Hardware constraints other than noise have been evaluated in [34][35][36][37]. Some of such constraints for a convolutional neural network on CIFAR dataset have been evaluated recently in [45].…”

A contextual analysis of multi-layer perceptron models in classifying hand-written digits and letters: limited resources

“…For such a non-convolutional architecture, the results are good enough and are in the expected range [19,8]. However, we do use a lot of memory space, just for training we have a 2D vector of shape 100k for the first dimension and 784 for the second dimension.…”

“…Experiments with different variants of MLP in [19,8] set a literature expectation for this task and dataset asserting the performance on different variations of MLP.…”

“…In [19], they apply Hardware-aware training algorithms for this task. However, the architecture of the multi-layer perceptron model is a basic one -3-layered neural network which seems similar to ours (before any processing step was applied).…”