Towards Efficient Neuromorphic Hardware: Unsupervised Adaptive Neuron Pruning

Guo, Wenzhe; Yantır, Hasan Erdem; Fouda, Mohammed E.; Eltawil, Ahmed M.; Salama, Khaled N.

doi:10.3390/electronics9071059

Cited by 17 publications

(15 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This network has been widely adopted to study the performance of a biological SNN for different purposes, such as studying different STDP models ( Diehl and Cook, 2015 ), improving STDP learning ( Panda et al, 2018 ), and pruning ( Shi et al, 2019 ; Guo et al, 2020b ).…”

Section: Comparison and Discussionmentioning

confidence: 99%

“…• In the literature, among unsupervised two-layer SNNs, this network structure shows the best performance with an STDP learning rule only (Shi et al, 2019;Tavanaei et al, 2019). • This network has been widely adopted to study the performance of a biological SNN for different purposes, such as studying different STDP models (Diehl and Cook, 2015), improving STDP learning (Panda et al, 2018), and pruning (Shi et al, 2019;Guo et al, 2020b Most importantly, this study provides important understanding of the impact of different coding schemes on various aspects of the performance of a neuromorphic system. With this understanding, it is beneficial for neuromorphic system researchers to consider and select corresponding coding schemes to achieve specific design goals.…”

Section: Comparison and Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Neural Coding in Spiking Neural Networks: A Comparative Study for Robust Neuromorphic Systems

et al. 2021

Self Cite

View full text Add to dashboard Cite

Various hypotheses of information representation in brain, referred to as neural codes, have been proposed to explain the information transmission between neurons. Neural coding plays an essential role in enabling the brain-inspired spiking neural networks (SNNs) to perform different tasks. To search for the best coding scheme, we performed an extensive comparative study on the impact and performance of four important neural coding schemes, namely, rate coding, time-to-first spike (TTFS) coding, phase coding, and burst coding. The comparative study was carried out using a biological 2-layer SNN trained with an unsupervised spike-timing-dependent plasticity (STDP) algorithm. Various aspects of network performance were considered, including classification accuracy, processing latency, synaptic operations (SOPs), hardware implementation, network compression efficacy, input and synaptic noise resilience, and synaptic fault tolerance. The classification tasks on Modified National Institute of Standards and Technology (MNIST) and Fashion-MNIST datasets were applied in our study. For hardware implementation, area and power consumption were estimated for these coding schemes, and the network compression efficacy was analyzed using pruning and quantization techniques. Different types of input noise and noise variations in the datasets were considered and applied. Furthermore, the robustness of each coding scheme to the non-ideality-induced synaptic noise and fault in analog neuromorphic systems was studied and compared. Our results show that TTFS coding is the best choice in achieving the highest computational performance with very low hardware implementation overhead. TTFS coding requires 4x/7.5x lower processing latency and 3.5x/6.5x fewer SOPs than rate coding during the training/inference process. Phase coding is the most resilient scheme to input noise. Burst coding offers the highest network compression efficacy and the best overall robustness to hardware non-idealities for both training and inference processes. The study presented in this paper reveals the design space created by the choice of each coding scheme, allowing designers to frame each scheme in terms of its strength and weakness given a designs’ constraints and considerations in neuromorphic systems.

show abstract

Section: Comparison and Discussionmentioning

confidence: 99%

Section: Comparison and Discussionmentioning

confidence: 99%

Neural Coding in Spiking Neural Networks: A Comparative Study for Robust Neuromorphic Systems

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, directly removing neurons from the network could cause severe deterioration of network performance. We compared the proposed online weight pruning methods with an online adaptive neuron pruning method presented in our previous work ( Guo et al, 2020 ). The comparison is shown in Figure 13 .…”

Section: Comparisons and Discussionmentioning

confidence: 99%

“…This bit can be simply attached to the weight bits in the memory with very little overhead. The number of NAND gates for an SNN with 100 neurons was estimated according to the proposed digital implementation from Guo et al (2020) . Clearly, the number of equivalent NAND gates for the pruning algorithm is much smaller than that for the SNN.…”

Section: Comparisons and Discussionmentioning

confidence: 99%

“…The estimated number of different phases in the table is for single-batch pruning. The number of clock cycles for the SNN training phase is much larger than the number of training SOPs/image, 96,099, since The number of NAND gates and BRAMs for an SNN were obtained with 100 neurons according to the proposed digital implementation from (Guo et al, 2020).…”

Section: Implementation Overheadmentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems

et al. 2020

Self Cite

View full text Add to dashboard Cite

To tackle real-world challenges, deep and complex neural networks are generally used with a massive number of parameters, which require large memory size, extensive computational operations, and high energy consumption in neuromorphic hardware systems. In this work, we propose an unsupervised online adaptive weight pruning method that dynamically removes non-critical weights from a spiking neural network (SNN) to reduce network complexity and improve energy efficiency. The adaptive pruning method explores neural dynamics and firing activity of SNNs and adapts the pruning threshold over time and neurons during training. The proposed adaptation scheme allows the network to effectively identify critical weights associated with each neuron by changing the pruning threshold dynamically over time and neurons. It balances the connection strength of neurons with the previous layer with adaptive thresholds and prevents weak neurons from failure after pruning. We also evaluated improvement in the energy efficiency of SNNs with our method by computing synaptic operations (SOPs). Simulation results and detailed analyses have revealed that applying adaptation in the pruning threshold can significantly improve network performance and reduce the number of SOPs. The pruned SNN with 800 excitatory neurons can achieve a 30% reduction in SOPs during training and a 55% reduction during inference, with only 0.44% accuracy loss on MNIST dataset. Compared with a previously reported online soft pruning method, the proposed adaptive pruning method shows 3.33% higher classification accuracy and 67% more reduction in SOPs. The effectiveness of our method was confirmed on different datasets and for different network sizes. Our evaluation showed that the implementation overhead of the adaptive method regarding speed, area, and energy is negligible in the network. Therefore, this work offers a promising solution for effective network compression and building highly energy-efficient neuromorphic systems in real-time applications.

show abstract

Fault-Tolerant Spiking Neural Network Mapping Algorithm and Architecture to 3D-NoC-Based Neuromorphic Systems

Yerima,

Ikechukwu,

Dang

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Neuromorphic computing uses spiking neuron network models to solve machine learning problems in a more energy-efficient way when compared to conventional artificial neural networks. However, mapping the various network components to the neuromorphic hardware is not trivial to realize the desired model for an actual simulation. Moreover, neurons and synapses could be affected by noise due to external interference or random actions of other components (i.e., neurons), which eventually lead to unreliable results. This work proposes a fault-tolerant spiking neural network mapping algorithm and architecture to a 3D network-on-chip (NoC)-based neuromorphic system (R-NASH-II) based on a rank and selection mapping mechanism (RSM). The RSM allows the ranking and rapid selection of neurons for fault-tolerant mapping. Evaluation results show that with our proposed mechanism, we could maintain a mapping efficiency of 100% with 20% spare rate and a fault rate (40%) more than in the previous mapping framework. The Monte Carlo simulation evaluation of reliability shows that the RSM mechanism has increased the mean time to failure (MTTF) of the previous mapping technique by 43% on average. Furthermore, the operational availability of the RSM for mapping to a 4 × 4 × 4 (smallest) and 6 × 6 × 6 (largest) NoC is 88% and 67% respectively.

show abstract

Towards Efficient Neuromorphic Hardware: Unsupervised Adaptive Neuron Pruning

Cited by 17 publications

References 22 publications

Neural Coding in Spiking Neural Networks: A Comparative Study for Robust Neuromorphic Systems

Neural Coding in Spiking Neural Networks: A Comparative Study for Robust Neuromorphic Systems

Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems

Fault-Tolerant Spiking Neural Network Mapping Algorithm and Architecture to 3D-NoC-Based Neuromorphic Systems

Contact Info

Product

Resources

About