Reducing Latency in a Converted Spiking Video Segmentation Network

Cheni, Qinyu; Rueckauer, Bodo; Li, Li; Delbrück, Tobi; Liu, Shih-Chii

doi:10.1109/iscas51556.2021.9401667

Cited by 12 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Conventionally, there are two distinct routes to train a functional SNN model: (1) training SNN from scratch; (2) converting a pretrained DNN to SNN. Methods for direct training of SNNs have made tremendous progress recently, but are often computationally expensive and difficult to scale up to some of the more challenging tasks [48]. Conversion of pretrained DNNs to SNNs is a more straightforward approach to achieve an SNN with competitive state-of-the-art accuracy to the DNN on a particular task.…”

Section: Theoretical Model Of Snnmentioning

confidence: 99%

Ultralow-power spiking neural networks for 1024-ary orbital angular momentum shift keying free-space optical communication

Chen

et al. 2023

J. Opt.

View full text Add to dashboard Cite

The theoretical unbounded orbital angular momentum (OAM) states can be exploited as data bits in the OAM shift keying (OAM-SK) free-space optical (FSO) communications. In order to cope with the atmospheric turbulence (AT) and misalignment in practical applications, various machine learning algorithms, or neural networks (NNs), have been put forward to decode the OAM states. However, to recognize the hybrid spatial modes representing a large bit states, the massive learnable nodes, longer computation time and more training parameters are required to improve the capability of the NNs, resulting in energy efficiency burden to the hardware device. In this paper, the event-based Spiking Neural Network (SNN) is utilized to recognize the hybrid spatial modes consisting of superposed coaxial LG modes with l ranging from 0 to 9 and p=0, which is termed as Spiking OAM-recognition neural network (Spiking-ORNN). In comparison to the previous solution of running deep neural networks (DNNs) on graphics processing units (GPUs), the neuromorphic solution of running Spiking-ORNN on neuromorphic chips exhibits 4300x higher energy efficiency without obvious sacrifice of recognition accuracy (less than 0.5%). Moreover, we experimentally demonstrate a 10-meter 1024-ary OAM-SK FSO communication for the transmission of an image with a 10-bit grey level, wherein the peak signal-to-noise ratio of the received image can exceed 41.4 dB under the AT of =1e−15 m−2/3. We anticipate that our results can stimulate further researches on the utilization of the brain-like SNN chips to reduce the energy consumptions based on the artificial-intelligence-enhanced optoelectronic systems.

show abstract

Section: Theoretical Model Of Snnmentioning

confidence: 99%

Ultralow-power spiking neural networks for 1024-ary orbital angular momentum shift keying free-space optical communication

Chen

et al. 2023

J. Opt.

View full text Add to dashboard Cite

show abstract

“…The training of SNNs is mainly divided into the following three categories: gradient backpropagation-based methods (Wu et al 2018(Wu et al , 2019Zheng et al 2021;Shen, Zhao, and Zeng 2022a;Li et al 2022c;Deng et al 2022), spiking time-dependent plasticity (STDP)-based methods (Diehl and Cook 2015;Hao et al 2020;Zhao et al 2020;Dong et al 2022), and conversion-based methods (Han, Srinivasan, and Roy 2020;Bu et al 2021;Li and Zeng 2022;Liu et al 2022;Li et al 2022b). With these proposed algorithms, SNNs show excellent performance in various complex scenarios (Stagsted et al 2020;Godet et al 2021;Sun, Zeng, and Zhang 2021;Cheni et al 2021). In particular, SNNs have shown promising results in processing neuromorphic, event-based data due to their ability to process information in the time dimension (Xing, Di Caterina, and Soraghan 2020;Chen et al 2020;Viale et al 2021).…”

Section: Introductionmentioning

confidence: 99%

An Efficient Knowledge Transfer Strategy for Spiking Neural Networks from Static to Event Domain

He,

Zhao,

et al. 2024

AAAI

View full text Add to dashboard Cite

Spiking neural networks (SNNs) are rich in spatio-temporal dynamics and are suitable for processing event-based neuromorphic data. However, event-based datasets are usually less annotated than static datasets. This small data scale makes SNNs prone to overfitting and limits their performance. In order to improve the generalization ability of SNNs on event-based datasets, we use static images to assist SNN training on event data. In this paper, we first discuss the domain mismatch problem encountered when directly transferring networks trained on static datasets to event data. We argue that the inconsistency of feature distributions becomes a major factor hindering the effective transfer of knowledge from static images to event data. To address this problem, we propose solutions in terms of two aspects: feature distribution and training strategy. Firstly, we propose a knowledge transfer loss, which consists of domain alignment loss and spatio-temporal regularization. The domain alignment loss learns domain-invariant spatial features by reducing the marginal distribution distance between the static image and the event data. Spatio-temporal regularization provides dynamically learnable coefficients for domain alignment loss by using the output features of the event data at each time step as a regularization term. In addition, we propose a sliding training strategy, which gradually replaces static image inputs probabilistically with event data, resulting in a smoother and more stable training for the network. We validate our method on neuromorphic datasets, including N-Caltech101, CEP-DVS, and N-Omniglot. The experimental results show that our proposed method achieves better performance on all datasets compared to the current state-of-the-art methods. Code is available at https://github.com/Brain-Cog-Lab/Transfer-for-DVS.

show abstract

“…However, it might cause decreased accuracy when outdated information from the previous frame is carried over. To capitalize on the useful initialization, we previously proposed an adaptable interval reset of the network state [6], which can achieve 35x speedup with acceptable accuracy loss.…”

Section: Introductionmentioning

confidence: 99%

FrameFire: Enabling Efficient Spiking Neural Network Inference for Video Segmentation

Chen¹,

Sun²,

Gao³

et al. 2023

2023 IEEE 5th International Conference on Artificial Intelligence Circuits and Systems (AICAS)

View full text Add to dashboard Cite

Fast video recognition is essential for real-time scenarios, e.g., autonomous driving. However, applying existing Deep Neural Networks (DNNs) to individual high-resolution images is expensive due to large model sizes. Spiking Neural Networks (SNNs) are developed as a promising alternative to DNNs due to their more realistic brain-inspired computing models. SNNs have sparse neuron firing over time, i.e., spatio-temporal sparsity; thus they are useful to enable energy-efficient computation. However, exploiting the spatio-temporal sparsity of SNNs in hardware leads to unpredictable and unbalanced workloads, degrading energy efficiency. In this work, we, therefore, propose an SNN accelerator called FrameFire for efficient video processing. We introduce a Keyframe-dominated Workload Balance Schedule (KWBS) method. It accelerates the image recognition network with sparse keyframes, then records and analyzes the current workload distribution on hardware to facilitate scheduling workloads in subsequent frames. FrameFire is implemented on a Xilinx XC7Z035 FPGA and verified by video segmentation tasks. The results show that the throughput is improved by 1.7× with the KWBS method. FrameFire achieved 1.04 KFPS throughput and 1.15 mJ/frame recognition energy.

show abstract

Reducing Latency in a Converted Spiking Video Segmentation Network

Cited by 12 publications

References 17 publications

Ultralow-power spiking neural networks for 1024-ary orbital angular momentum shift keying free-space optical communication

Ultralow-power spiking neural networks for 1024-ary orbital angular momentum shift keying free-space optical communication

An Efficient Knowledge Transfer Strategy for Spiking Neural Networks from Static to Event Domain

FrameFire: Enabling Efficient Spiking Neural Network Inference for Video Segmentation

Contact Info

Product

Resources

About