Enhancing spiking neural networks with hybrid top-down attention

Abstract

As the representatives of brain-inspired models at the neuronal level, spiking neural networks (SNNs) have shown great promise in processing spatiotemporal information with intrinsic temporal dynamics. SNNs are expected to further improve their robustness and computing efficiency by introducing top-down attention at the architectural level, which is crucial for the human brain to support advanced intelligence. However, this attempt encounters difficulties in optimizing the attention in SNNs largely due to the lack of annotations. Here, we develop a hybrid network model with a top-down attention mechanism (HTDA) by incorporating an artificial neural network (ANN) to generate attention maps based on the features extracted by a feedforward SNN. The attention map is then used to modulate the encoding layer of the SNN so that it focuses on the most informative sensory input. To facilitate direct learning of attention maps and avoid labor-intensive annotations, we propose a general principle and a corresponding weakly-supervised objective, which promotes the HTDA model to utilize an integral and small subset of the input to give accurate predictions. On this basis, the ANN and the SNN can be jointly optimized by surrogate gradient descent in an end-to-end manner. We comprehensively evaluated the HTDA model on object recognition tasks, which demonstrates strong robustness to adversarial noise, high computing efficiency, and good interpretability. On the widely-adopted CIFAR-10, CIFAR-100, and MNIST benchmarks, the HTDA model reduces firing rates by up to 50% and improves adversarial robustness by up to 10% with comparable or better accuracy compared with the state-of-the-art SNNs. The HTDA model is also verified on dynamic neuromorphic datasets and achieves consistent improvements. This study provides a new way to boost the performance of SNNs by employing a hybrid top-down attention mechanism.

Advancing brain-inspired computing with hybrid neural networks

“…(a) The hybrid reasoning network [ 24 ] with serial structure for multistage robust question answering. (b) The hybrid top-down attention network [ 56 ] with feedback structure for multilevel efficient perception. (c) Hybrid sensing network [ 24 ] with parallel structure for multipathway tracking.…”

“…Specifically, in the context of visual processing, high-level information obtained through feature extraction can effectively regulate the operations of the front-end network. Evidencing the principle of feedback structures is the hybrid top-down attention network [ 56 ], which combines a feedforward SNN and a feedback ANN to effectuate a form of top-down attention mechanism. The ANN generates attention maps based on extracted features from the SNN, thereby modulating the encoding layer within the SNN.…”

Spiking NeRF: Representing the Real-World Geometry by a Discontinuous Representation

“…Although previous works have demonstrated SNN applications on a wide range of tasks, they are still limited in their performance (Li et al 2021b;Fang et al 2021). Meanwhile, there has been a growing interest in exploring the potential benefits of combining ANNs and SNNs (Kugele et al 2021;Liu and Zhao 2022).…”