Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization

Panda, Priyadarshini; Aketi, Sai Aparna; Roy, Kaushik

doi:10.3389/fnins.2020.00653

Cited by 93 publications

(65 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Taking inspiration from the extremely efficient brain, spiking neural networks (SNNs) [22] combine binary valued activations (spikes) with asynchronous and sparse communication. Such SNNs are arguably also more hardware friendly [10] and energy-efficient [28]. However, compared to ANNs, the development of SNNs is in its early phase.…”

Section: Introductionmentioning

confidence: 99%

Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks

Yin

Corradi

Bohté

2020

International Conference on Neuromorphic Systems 2020

View full text Add to dashboard Cite

The emergence of brain-inspired neuromorphic computing as a paradigm for edge AI is motivating the search for high-performance and efficient spiking neural networks to run on this hardware. However, compared to classical neural networks in deep learning, current spiking neural networks lack competitive performance in compelling areas. Here, for sequential and streaming tasks, we demonstrate how a novel type of adaptive spiking recurrent neural network (SRNN) is able to achieve state-of-the-art performance compared to other spiking neural networks and almost reach or exceed the performance of classical recurrent neural networks (RNNs) while exhibiting sparse activity. From this, we calculate a >100x energy improvement for our SRNNs over classical RNNs on the harder tasks. To achieve this, we model standard and adaptive multiple-timescale spiking neurons as self-recurrent neural units, and leverage surrogate gradients and auto-differentiation in the PyTorch Deep Learning framework to efficiently implement backpropagation-through-time, including learning of the important spiking neuron parameters to adapt our spiking neurons to the tasks. CCS CONCEPTS• Computing methodologies → Neural networks; Supervised learning by classification; Continuous models.

show abstract

Section: Introductionmentioning

confidence: 99%

Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks

Yin

Corradi

Bohté

2020

International Conference on Neuromorphic Systems 2020

View full text Add to dashboard Cite

show abstract

“…Moreover, it is well-known that the event-driven nature of SNN assists in reducing network activation which in turn reduced energy dissipation of computation. Using method presented in Panda et al ( 2020 ), we compute the energy advantage of H-SNN over DNN baselines during inference as follows: 1.0 for H-SNN, 1.55 for 3D CNN-α, and 3.37 for 3D ShuffleNetV2; 3D MobileNetV2, 3D CNN-β, and CNN+LSTM consumes 9.01×, 11.80× and 28.88× higher energy than H-SNN, respectively. BP-SNN and BP-SNN-LS uses similar energy as H-SNN.…”

Section: Resultsmentioning

confidence: 99%

A Heterogeneous Spiking Neural Network for Unsupervised Learning of Spatiotemporal Patterns

et al. 2021

View full text Add to dashboard Cite

This paper introduces a heterogeneous spiking neural network (H-SNN) as a novel, feedforward SNN structure capable of learning complex spatiotemporal patterns with spike-timing-dependent plasticity (STDP) based unsupervised training. Within H-SNN, hierarchical spatial and temporal patterns are constructed with convolution connections and memory pathways containing spiking neurons with different dynamics. We demonstrate analytically the formation of long and short term memory in H-SNN and distinct response functions of memory pathways. In simulation, the network is tested on visual input of moving objects to simultaneously predict for object class and motion dynamics. Results show that H-SNN achieves prediction accuracy on similar or higher level than supervised deep neural networks (DNN). Compared to SNN trained with back-propagation, H-SNN effectively utilizes STDP to learn spatiotemporal patterns that have better generalizability to unknown motion and/or object classes encountered during inference. In addition, the improved performance is achieved with 6x fewer parameters than complex DNNs, showing H-SNN as an efficient approach for applications with constrained computation resources.

show abstract

“…With increasing preponderance in event-based algorithms, data-driven approaches are mainly categorized into two classes: asynchronous spiking neural networks [46,49,52,54] and standard learning architectures [7,40,65,68]. As bio-inspired methods, SNNs offer asynchronous inferences at a fraction of power consumption but suffer from the vanishing spike phenomenon [76] in deeper layers. In contrast, the latter methods commonly trade-off efficiency for accuracy and generalize well in complex vision tasks, sacrificing the inherent sparsity of spatio-temporal events.…”

Section: Data-driven Approachesmentioning

confidence: 99%

“…Basically, existing event-based algorithms either refer to frame-based computer vision techniques or take inspiration from biological systems. However, pure bioinspired SNN architectures suffer from the vanishing spike phenomenon [76] in deeper layers and the lack of specific hardware. As for standard learning architectures designed for frame-based vision tasks, the reliance of intermediate image-like representation usually leads to the loss of asynchronous property and redundant computation.…”

Section: Algorithmsmentioning

confidence: 99%

Data‐Driven Technology in Event‐Based Vision

Sun

Shi²,

Zhang³

et al. 2021

Complexity

View full text Add to dashboard Cite

Event cameras which transmit per-pixel intensity changes have emerged as a promising candidate in applications such as consumer electronics, industrial automation, and autonomous vehicles, owing to their efficiency and robustness. To maintain these inherent advantages, the trade-off between efficiency and accuracy stands as a priority in event-based algorithms. Thanks to the preponderance of deep learning techniques and the compatibility between bio-inspired spiking neural networks and event-based sensors, data-driven approaches have become a hot spot, which along with the dedicated hardware and datasets constitute an emerging field named event-based data-driven technology. Focusing on data-driven technology in event-based vision, this paper first explicates the operating principle, advantages, and intrinsic nature of event cameras, as well as background knowledge in event-based vision, presenting an overview of this research field. Then, we explain why event-based data-driven technology becomes a research focus, including reasons for the rise of event-based vision and the superiority of data-driven approaches over other event-based algorithms. Current status and future trends of event-based data-driven technology are presented successively in terms of hardware, datasets, and algorithms, providing guidance for future research. Generally, this paper reveals the great prospects of event-based data-driven technology and presents a comprehensive overview of this field, aiming at a more efficient and bio-inspired visual system to extract visual features from the external environment.

show abstract

Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization

Cited by 93 publications

References 48 publications

Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks

Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks

A Heterogeneous Spiking Neural Network for Unsupervised Learning of Spatiotemporal Patterns

Data‐Driven Technology in Event‐Based Vision

Contact Info

Product

Resources

About