Advantages of heterogeneity of parameters in spiking neural network training

Perez-Nieves, Nicolas; Leung, Vincent C. H.; Dragotti, Pier Luigi; Goodman, Dan F. M.

doi:10.32470/ccn.2019.1173-0

Cited by 6 publications

(8 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Consider also the Fast Sigmoid surrogate gradient (Zenke & Ganguli, 2018;Perez-Nieves & Goodman, 2021) that avoids computing the exponential function in Sigmoid to obtain the gradient:…”

Section: A Proofs Of Theoretical Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Energy Efficient Training of SNN using Local Zeroth Order Method

Mukhoty¹,

Bojković²,

Vazelhes³

et al. 2023

Preprint

View full text Add to dashboard Cite

Spiking neural networks are becoming increasingly popular for their low energy requirement in real-world tasks with accuracy comparable to the traditional ANNs. SNN training algorithms face the loss of gradient information and nondifferentiability due to the Heaviside function in minimizing the model loss over model parameters.To circumvent the problem surrogate method uses a differentiable approximation of the Heaviside in the backward pass, while the forward pass uses the Heaviside as the spiking function. We propose to use the zeroth order technique at the neuron level to resolve this dichotomy and use it within the automatic differentiation tool. As a result, we establish a theoretical connection between the proposed local zeroth-order technique and the existing surrogate methods and vice-versa. The proposed method naturally lends itself to energyefficient training of SNNs on GPUs. Experimental results with neuromorphic datasets show that such implementation requires less than 1% neurons to be active in the backward pass, resulting in a 100x speed-up in the backward computation time. Our method offers better generalization compared to the state-of-the-art energy-efficient technique while maintaining similar efficiency.

show abstract

“…Consider also the Fast Sigmoid surrogate gradient (Zenke & Ganguli, 2018;Perez-Nieves & Goodman, 2021) that avoids computing the exponential function in Sigmoid to obtain the gradient:…”

Section: A Proofs Of Theoretical Resultsmentioning

confidence: 99%

“…We implement the technique for fully connected networks, with two hidden layers. Implementing the method for deeper networks should be straightforward as shown in (Perez-Nieves & Goodman, 2021). However, we leave the adaptation of the method with convolutional layers for the future work.…”

Section: Discussionmentioning

confidence: 99%

Energy Efficient Training of SNN using Local Zeroth Order Method

Mukhoty¹,

Bojković²,

Vazelhes³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In summary, by leveraging a novel spiking RNN model with in vivo recordings, we have shown that the heterogeneous neural response profiles widely observed during behavior are constrained by local synaptic structures shaped by spike-timing dependent plasticity mechanisms. Our model sits at the nexus between two recent trends in neural network modeling: First, recent work has successfully extended general-purpose learning algorithms (i.e., FORCE) designed for rate-based networks to networks with spiking units 35,47–49 . Second, there has been renewed interest in using RNNs to understand the role of biophysically-motivated synaptic plasticity rules in the formation of stable neural assemblies for memory storage and retrieval 32,33,50 .…”

Section: Discussionmentioning

confidence: 99%

Contributions and synaptic basis of diverse cortical neuron responses to task performance

Insanally

Albanna

Toth

et al. 2022

Preprint

View full text Add to dashboard Cite

Neuronal responses during behavior are diverse, ranging from highly reliable 'classical' responses to irregular or seemingly-random 'non-classically responsive' firing. While a continuum of response properties is frequently observed across neural systems, little is known about the synaptic origins and contributions of diverse response profiles to network function, perception, and behavior. Here we use a task-performing, spiking recurrent neural network model incorporating spike-timingdependent plasticity that captures heterogeneous responses measured from auditory cortex of behaving rodents. Classically responsive and non-classically responsive model units contributed to task performance via output and recurrent connections, respectively. Excitatory and inhibitory plasticity independently shaped spiking responses and task performance. Local patterns of synaptic inputs predicted spiking response properties of network units as well as the responses of auditory cortical neurons from in vivo whole-cell recordings during behavior. Thus a diversity of neural response profiles emerges from synaptic plasticity rules with distinctly important functions for network performance.

show abstract

“…SNN-IIR [26] is proposed by Fang et al to search for the optimal synapse filter kernels and weights for SNN to learn the spatio-temporal patterns. Nicolas et al [30] propose a sparse backpropagation method for SNNs that is faster and more memory efficient.…”

Section: Related Workmentioning

confidence: 99%

“…Different voxel grids are selected for various datasets, more in detail, 2048, 512, 512, 512 are chosen for the DVS128-Gait-Day, ASL-DVS, N-MNIST and HARDVS datasets. After considering the spatiotemporal discrepancy across different datasets, we set the scale (v h , v w , v t ) of voxel grid as (10,10,10) for ASL-DVS, (4, 4, 4) for DVS128-Gait-Day, (20, 2, 2), (50,30,20) for N-MNIST and HARDVS datasets. When building graphs for the voxel branch, the threshold R is set as 2.…”

Section: B Implementation Detailsmentioning

confidence: 99%

Data Representation and Learning With Graph Diffusion-Embedding Networks

Jiang

Lin

Tang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Considering the balance of performance and efficiency, sampled point and voxel methods are usually employed to down-sample dense events into sparse ones. After that, one popular way is to leverage a graph model which treats the sparse points/voxels as nodes and adopts graph neural networks (GNNs) to learn the representation for event data. Although good performance can be obtained, however, their results are still limited mainly due to two issues. (1) Existing event GNNs generally adopt the additional max (or mean) pooling layer to summarize all node embeddings into a single graph-level representation for the whole event data representation. However, this approach fails to capture the importance of graph nodes and also fail to be fully aware of the node representations. (2) Existing methods generally employ either a sparse point or voxel graph representation model which thus lacks consideration of the complementary between these two types of representation models. To address these issues, in this paper, we propose a novel dual point-voxel absorbing graph representation learning for event stream data representation. To be specific, given the input event stream, we first transform it into the sparse event cloud and voxel grids and build dual absorbing graph models for them respectively. Then, we design a novel absorbing graph convolutional network (AGCN) for our dual absorbing graph representation and learning. The key aspect of the proposed AGCN is its ability to effectively capture the importance of nodes and thus be fully aware of node representations in summarizing all node representations through the introduced absorbing nodes. Finally, the event representations of dual learning branches are concatenated together to extract the complementary information of two cues. The output is then fed into a linear layer for event data classification. Extensive experiments on multiple event-based classification benchmark datasets fully validated the effectiveness of our framework. New state-of-the-art performances are achieved on the ASL-DVS (99.7) and DVS128-Gait-Day (99.7) datasets. Both the source code and pre-trained models will be released.

show abstract

Advantages of heterogeneity of parameters in spiking neural network training

Cited by 6 publications

References 4 publications

Energy Efficient Training of SNN using Local Zeroth Order Method

Energy Efficient Training of SNN using Local Zeroth Order Method

Contributions and synaptic basis of diverse cortical neuron responses to task performance

Data Representation and Learning With Graph Diffusion-Embedding Networks

Contact Info

Product

Resources

About