Attention Spiking Neural Networks

Yao, Man; Zhao, Guangshe; Zhang, Hengyu; Hu, Yifan; Deng, Lei; Xu, Bo; Li, Guoqi

doi:10.1109/tpami.2023.3241201

Cited by 70 publications

(33 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Further, this is achieved by utilizing merely around 20% of the computation (in FLOPs) required by the SOTA methods. Furthermore, our proposed method also outperforms SOTA SNN baselines [24], [46], [47] in this regression task of human pose tracking.…”

mentioning

confidence: 75%

“…To avoid confusion, it is important to note that the spiking transformers presented in [21], [58] are not SNN-based transformers, but rather ANN-based or mixed models. The two recent works [46], [47] are most related to our proposed spiking spatiotemporal transformer. In MA-SNN [46], multi-dimensional attention is proposed in an SNNs framework, yet this attention is instead based on real values of membrane potentials, thus in a sense violating the efficiency design of SNNs.…”

Section: Related Workmentioning

confidence: 99%

“…The two recent works [46], [47] are most related to our proposed spiking spatiotemporal transformer. In MA-SNN [46], multi-dimensional attention is proposed in an SNNs framework, yet this attention is instead based on real values of membrane potentials, thus in a sense violating the efficiency design of SNNs. In contrast, the spatiotemporal attention mechanism in our model is entirely based on spike tensors, which is realised by the proposed Hamming similarity score.…”

Section: Related Workmentioning

confidence: 99%

“…EventCap [28] and EventHPE [29] are most recent works using event camera for human pose tracking. We also include three popular ANN models (ResNet-GRU [15], ResNet-TF [49] and Video-Swin [50]), one mixed model (SEW-ResNet-TF ) and four SNN models (ANN2SNN [39], SEW-ResNet [24], MA-SNN [46] and SpikeFormer [47]) as benchmarks to illustrate the effectiveness of our SNN approach with novel Spiking Spatiotemporal Transformer. Underline denotes the best value except EventHPE(GT).…”

Section: Tablementioning

confidence: 99%

“…SEW-ResNet [24] is the backbone used in our approach without Spiking Spatiotemporal Transformer. MA-SNN [46] represents multi-dimensional attention SNNs where SEW-ResNet50 is used for a fair comparison. SpikeFormer [47] indicates an SNN-based vision transformer (ViT) [62], where dot-product is directly adopted in the self-attention module.…”

Section: Comparison With Snn Modelsmentioning

confidence: 99%

See 4 more Smart Citations

EventHPE: Event-based 3D Human Pose and Shape Estimation

Zou

Guo

Zuo

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Event camera, as an emerging biologically-inspired vision sensor for capturing motion dynamics, presents new potential for 3D human pose tracking, or video-based 3D human pose estimation. However, existing works in pose tracking either require the presence of additional gray-scale images to establish a solid starting pose, or ignore the temporal dependencies all together by collapsing segments of event streams to form static event frames. Meanwhile, although the effectiveness of Artificial Neural Networks (ANNs, a.k.a. dense deep learning) has been showcased in many event-based tasks, the use of ANNs tends to neglect the fact that compared to the dense frame-based image sequences, the occurrence of events from an event camera is spatiotemporally much sparser. Motivated by the above mentioned issues, we present in this paper a dedicated end-to-end sparse deep learning approach for event-based pose tracking: 1) to our knowledge this is the first time that 3D human pose tracking is obtained from events only, thus eliminating the need of accessing to any frame-based images as part of input; 2) our approach is based entirely upon the framework of Spiking Neural Networks (SNNs), which consists of Spike-Element-Wise (SEW) ResNet and a novel Spiking Spatiotemporal Transformer; 3) a large-scale synthetic dataset is constructed that features a broad and diverse set of annotated 3D human motions, as well as longer hours of event stream data, named SynEventHPD. Empirical experiments demonstrate that, with superior performance over the state-of-the-art (SOTA) ANNs counterparts, our approach also achieves a significant computation reduction of 80% in FLOPS. Furthermore, our proposed method also outperforms SOTA SNNs in the regression task of human pose tracking. Our implementation is available at https://github.com/JimmyZou/HumanPoseTracking SNN and dataset will be released upon paper acceptance.

show abstract

mentioning

confidence: 75%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Tablementioning

confidence: 99%

Section: Comparison With Snn Modelsmentioning

confidence: 99%

See 3 more Smart Citations

EventHPE: Event-based 3D Human Pose and Shape Estimation

Zou

Guo

Zuo

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

Memristor‐Based Neuromorphic Chips

Duan,

Cao,

Gao

et al. 2024

Advanced Materials

View full text Add to dashboard Cite

In the era of information, characterized by an exponential growth in data volume and an escalating level of data abstraction, there has been a substantial focus on brain‐like chips, which are known for their robust processing power and energy‐efficient operation. Memristors are widely acknowledged as the optimal electronic devices for the realization of neuromorphic computing, due to their innate ability to emulate the interconnection and information transfer processes witnessed among neurons. This review paper focuses on memristor‐based neuromorphic chips, which provide an extensive description of the working principle and characteristic features of memristors, along with their applications in the realm of neuromorphic chips. Subsequently, a thorough discussion of the memristor array, which serves as the pivotal component of the neuromorphic chip, as well as an examination of the present mainstream neural networks, is delved. Furthermore, the design of the neuromorphic chip is categorized into three crucial sections, including synapse‐neuron cores, networks on chip (NoC), and neural network design. Finally, the key performance metrics of the chip is highlighted, as well as the key metrics related to the memristor devices are employed to realize both the synaptic and neuronal components.

show abstract