Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations

Abstract

In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps∕s∕W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.

Bridging Biological and Artificial Neural Networks with Emerging Neuromorphic Devices: Fundamentals, Progress, and Challenges

“…[3,[27][28][29][30] In the hardware implementation of FC networks, e.g., multi-layer perceptrons (MLPs), the weight matrices could be directly mapped to the conductance matrices of memristive crossbar arrays. [3,[27][28][29][30] In the hardware implementation of FC networks, e.g., multi-layer perceptrons (MLPs), the weight matrices could be directly mapped to the conductance matrices of memristive crossbar arrays.…”

“…[3,30] The second approach is to use pulse trains continuously program the target device until the conductance change reaches the desired value before programming the next one (sequential update). For practical conductance update, mainly three approaches have been developed in literature.…”

“…[3] Experimentally, the first attempt to build an artificial neural machine is SNARC, short for "Stochastic Neural-Analog Reinforcement Computer," using vacuum tubes to simulate a network of 40 neurons, dated back to the 1950s. For example, it is projected that a new architecture of resistive processing unit (RPU) could provide 30 000× acceleration compared to state-of-the-art CPU/ GPU in training deep neural networks (DNNs).…”

Computing of temporal information in spiking neural networks with ReRAM synapses

“…[19][20][21] The major issue for this approach is the non-linear weight update and the large variability of resistive switching devices. 20,22 On the other hand, brain-inspired spiking neural networks (SNNs) aim at replicating the brain structure and computation in hardware. Learning usually takes place via spike-timing dependent plasticity (STDP), [23][24][25][26][27] where synapses can update their weight according to the timing between spikes of the pre-synaptic neuron (PRE) and post-synaptic neuron (POST).…”

Structure and basic functions of cognitive neural network machine

“…Each wafer allows emulating 200,000 neurons and 49 million synaptic connections [3]. As an analog cognitive neural network, there is a resistive processor of IBM [4], and others [5].…”