Fast classification using sparsely active spiking networks

Mostafa, Hesham; Pedroni, Bruno U.; Sheik, Sadique; Cauwenberghs, Gert

doi:10.1109/iscas.2017.8050527

Cited by 39 publications

(36 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These approaches, however, are based on spiking networks with rate coding, which typically require more memory accesses and longer processing time for each training pattern compared to BSNs. An alternative training approach based on exact backpropagation and temporal coding in spiking networks (Mostafa, 2016 ; Mostafa et al, 2017 ) has been shown to lead to highly sparse spiking activity during training and inference, and could potentially be more energetically efficient than training BSNs using the approach presented in this paper.…”

Section: Conclusion and Discussionmentioning

confidence: 99%

Hardware-Efficient On-line Learning through Pipelined Truncated-Error Backpropagation in Binary-State Networks

et al. 2017

Self Cite

View full text Add to dashboard Cite

Artificial neural networks (ANNs) trained using backpropagation are powerful learning architectures that have achieved state-of-the-art performance in various benchmarks. Significant effort has been devoted to developing custom silicon devices to accelerate inference in ANNs. Accelerating the training phase, however, has attracted relatively little attention. In this paper, we describe a hardware-efficient on-line learning technique for feedforward multi-layer ANNs that is based on pipelined backpropagation. Learning is performed in parallel with inference in the forward pass, removing the need for an explicit backward pass and requiring no extra weight lookup. By using binary state variables in the feedforward network and ternary errors in truncated-error backpropagation, the need for any multiplications in the forward and backward passes is removed, and memory requirements for the pipelining are drastically reduced. Further reduction in addition operations owing to the sparsity in the forward neural and backpropagating error signal paths contributes to highly efficient hardware implementation. For proof-of-concept validation, we demonstrate on-line learning of MNIST handwritten digit classification on a Spartan 6 FPGA interfacing with an external 1Gb DDR2 DRAM, that shows small degradation in test error performance compared to an equivalently sized binary ANN trained off-line using standard back-propagation and exact errors. Our results highlight an attractive synergy between pipelined backpropagation and binary-state networks in substantially reducing computation and memory requirements, making pipelined on-line learning practical in deep networks.

show abstract

Section: Conclusion and Discussionmentioning

confidence: 99%

Hardware-Efficient On-line Learning through Pipelined Truncated-Error Backpropagation in Binary-State Networks

et al. 2017

Self Cite

View full text Add to dashboard Cite

show abstract

“…Rate or time coding have significantly different impacts on such overheads. Using rate coding, a larger amount of spikes, with respect to time coding, is sent through the network and the overall system power consumption drastically increases, as shown in [130]. On the other hand, rate coding usually employs Poisson spike trains and an individual spike timing error is thus of small consequence for the precision of the network.…”

Section: Reduced Communication Costmentioning

confidence: 99%

“…Many argue that the main advantage of SNNs is that for a given input data, due to the event driven nature and the thresholding of every units, the evaluation will usually not require every neuron of every layer to fire [130]. As such, computation is sparse and the evaluation of the network on the input data requires less operations compared to the ANNs' frame-driven approach, where evaluation of every neurons of the network is required for every input data.…”

Section: Leverage High Sparsity Of Dataflowmentioning

confidence: 99%

“…To perform operations on specialized processors, the algorithms are usually modified, to better fit the circuit capabilities, even if the hardware and software are co-designed. We thus focus on the works which resulted into neuromorphic circuit evaluation of SNN on MNIST [34,115,130,193,201]. We exclusively consider hardware-related data to give a comparison overview of the latest SNN accelerators targeting low power and potential embedding ( Table 2).…”

Section: Low Power Spiking Machine Learningmentioning

confidence: 99%

See 1 more Smart Citation

Spiking Neural Networks Hardware Implementations and Challenges

Bouvier

Valentian

Mesquida

et al. 2019

J. Emerg. Technol. Comput. Syst.

197

View full text Add to dashboard Cite

Neuromorphic computing is henceforth a major research field for both academic and industrial actors. As opposed to Von Neumann machines, brain-inspired processors aim at bringing closer the memory and the computational elements to efficiently evaluate machine-learning algorithms. Recently, Spiking Neural Networks, a generation of cognitive algorithms employing computational primitives mimicking neuron and synapse operational principles, have become an important part of deep learning. They are expected to improve the computational performance and efficiency of neural networks, but are best suited for hardware able to support their temporal dynamics. In this survey, we present the state of the art of hardware implementations of spiking neural networks and the current trends in algorithm elaboration from model selection to training mechanisms. The scope of existing solutions is extensive; we thus present the general framework and study on a case-by-case basis the relevant particularities. We describe the strategies employed to leverage the characteristics of these event-driven algorithms at the hardware level and discuss their related advantages and challenges.

show abstract

“…We also compare our work with a wider range of implementations, including custom ASIC chips [8,41,50,59], neural processing units [18], spiking neural networks [14,28,42], crossbar implementations [57], and CPU/GPU-based solutions of the DropConnect approach [58] (the most accurate approach for MNIST to date; data is measured via i7-5820K, 32GB DDR3 with Nvidia Titan). Fig.…”

Section: Comparison To Other Mnist Implementationsmentioning

confidence: 99%

A Mixed Signal Architecture for Convolutional Neural Networks

Lou

Pan²,

McGuinness

et al. 2019

J. Emerg. Technol. Comput. Syst.

View full text Add to dashboard Cite

Deep neural network (DNN) accelerators with improved energy and delay are desirable for meeting the requirements of hardware targeted for IoT and edge computing systems. Convolutional neural networks (CoNNs) belong to one of the most popular types of DNN architectures. is paper presents the design and evaluation of an accelerator for CoNNs. e system-level architecture is based on mixed-signal, cellular neural networks (CeNNs). Speci cally, we present (i) the implementation of di erent layers, including convolution, ReLU, and pooling, in a CoNN using CeNN, (ii) modi ed CoNN structures with CeNN-friendly layers to reduce computational overheads typically associated with a CoNN, (iii) a mixed-signal CeNN architecture that performs CoNN computations in the analog and mixed signal domain, and (iv) design space exploration that identi es what CeNN-based algorithm and architectural features fare best compared to existing algorithms and architectures when evaluated over common datasets -MNIST and CIFAR-10. Notably, the proposed approach can lead to 8.7× improvements in energy-delay product (EDP) per digit classi cation for the MNIST dataset at iso-accuracy when compared with the state-of-the-art DNN engine, while our approach could o er 4.3× improvements in EDP when compared to other network implementations for the CIFAR-10 dataset.When considering application-speci c hardware to support neural networks, it is important that said hardware can implement networks that can be extensible to a large class of networks, and solve a large collection of application-level problems. Deep neural networks (DNNs) represent a class of such networks and have demonstrated their strength in applications such as playing the game of Go [54], image and video analysis [32], target tracking [31], etc. In this paper, we use convolutional neural network (CoNN) as a case study for DNNs due to its general prevalence. CoNNs are computationally intensive, which could lead to high latency and energy for inference and even higher latency/energy for training. e focus of this paper is on developing a low energy/delay mixed-signal system based on cellular neural networks (CeNNs) for realizing CoNN.A Cellular Nonlinear/Neural Network (CeNN) is an analog computing architecture [11] that could be well suited for many information processing tasks. In a CeNN, identical processing units (called cells) process analog information in a concurrent manner. Interconnection between cells is typically local (i.e., nearest neighbor) and space-invariant. For spatio-temporal applications, CeNNs can o er vastly superior performance and power e ciency when compared to conventional von Neumann architectures [47,61]. Using "CeNNs for CoNN" allows the bulk of the computation associated with a CoNN to be performed in the analog domain. Sensed information could immediately be processed with no analog-to-digital conversion (ADC). Also, inference-based processing tasks can tolerate lower precision (e.g., Google's TPU employs 8-bit integer matrix multiplies [24]) typically associa...

show abstract

Fast classification using sparsely active spiking networks

Cited by 39 publications

References 14 publications

Hardware-Efficient On-line Learning through Pipelined Truncated-Error Backpropagation in Binary-State Networks

Hardware-Efficient On-line Learning through Pipelined Truncated-Error Backpropagation in Binary-State Networks

Spiking Neural Networks Hardware Implementations and Challenges

A Mixed Signal Architecture for Convolutional Neural Networks

Contact Info

Product

Resources

About