Convolutional networks for fast, energy-efficient neuromorphic computing

Abstract

Deep networks are now able to achieve human-level performance on a broad spectrum of recognition tasks. Independently, neuromorphic computing has now demonstrated unprecedented energy-efficiency through a new chip architecture based on spiking neurons, low precision synapses, and a scalable communication network. Here, we demonstrate that neuromorphic computing, despite its novel architectural primitives, can implement deep convolution networks that (i) approach state-of-the-art classification accuracy across eight standard datasets encompassing vision and speech, (ii) perform inference while preserving the hardware's underlying energy-efficiency and high throughput, running on the aforementioned datasets at between 1,200 and 2,600 frames/s and using between 25 and 275 mW (effectively >6,000 frames/s per Watt), and (iii) can be specified and trained using backpropagation with the same ease-of-use as contemporary deep learning. This approach allows the algorithmic power of deep learning to be merged with the efficiency of neuromorphic processors, bringing the promise of embedded, intelligent, brain-inspired computing one step closer.convolutional network | neuromorphic | neural network | TrueNorth T he human brain is capable of remarkable acts of perception while consuming very little energy. The dream of brain-inspired computing is to build machines that do the same, requiring high-accuracy algorithms and efficient hardware to run those algorithms. On the algorithm front, building on classic work on backpropagation (1), the neocognitron (2), and convolutional networks (3), deep learning has made great strides in achieving human-level performance on a wide range of recognition tasks (4). On the hardware front, building on foundational work on silicon neural systems (5), neuromorphic computing, using novel architectural primitives, has recently demonstrated hardware capable of running 1 million neurons and 256 million synapses for extremely low power (just 70 mW at real-time operation) (6). Bringing these approaches together holds the promise of a new generation of embedded, real-time systems, but first requires reconciling key differences in the structure and operation between contemporary algorithms and hardware. Here, we introduce and demonstrate an approach we call Eedn, energy-efficient deep neuromorphic networks, which creates convolutional networks whose connections, neurons, and weights have been adapted to run inference tasks on neuromorphic hardware.For structure, typical convolutional networks place no constraints on filter sizes, whereas neuromorphic systems can take advantage of blockwise connectivity that limits filter sizes, thereby saving energy because weights can now be stored in local on-chip memory within dedicated neural cores. Here, we present a convolutional network structure that naturally maps to the efficient connection primitives used in contemporary neuromorphic systems. We enforce this connectivity constraint by partitioning filters into multiple groups and yet maintain network integra...

Energy-efficient neural network chips approach human recognition capabilities

“…Driving goals are to carry both the astounding energy efficiency of computations in neural networks of the brain and their learning capability into future generations of electronic hardware. A realization of this dream has now come one step closer, as reported by Esser et al (1). The authors demonstrate that a very energy-efficient implementation of an artificial neural network (i.e., of a circuit that shares properties with networks of neurons in the brain) achieves almost the same performance as humans as shown on eight benchmark datasets for recognizing images and sounds.…”

“…A learning algorithm like that in Esser et al (1) cannot do that. However, many online learning methods have been developed in machine learning.…”

“…The learning algorithm of Esser et al (1), which was independently discovered by Courbariaux and Bengio (6), applies another information compression trick proposed by Courbariaux et al (7) and Esser et al (8): after all iterations of the backprop algorithm are completed, one rounds the values of weights to just two or three values. Obviously, binary or ternary weights can be stored more efficiently on a chip than analog weights.…”

“…However, to make the backprop algorithm work one needs to store and update somewhere else precise analog weights during the iterative error reduction via gradient descent. Therefore, the learning process itself cannot be carried out on the chip in the approach of Esser et al (1). Another clever trick from Courbariaux et al (7) and Esser et al (8) is used to make the results of this external backprop algorithm applicable to the McCullochPitts neurons on the chip, despite the nondifferentiability of their activation function f: one replaces the true derivative of f, which has a singularity at 0, by the derivative of a virtual smooth activation function.…”

TrueHappiness: Neuromorphic emotion recognition on TrueNorth

“…We call this approach "train-then-constrain", comparing it to the "constrain-then-train" used in [13]. Using the "linear reset" mode, TrueNorth neurons can be configured to produce spiking rates similar to ReLUs.…”