Event-based Video Reconstruction via Potential-assisted Spiking Neural Network

Zhu, Lin; Wang, Xiao; Yi, Chuho; Li, Jianing; Huang, Tiejun

doi:10.1109/cvpr52688.2022.00358

Cited by 75 publications

(33 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Frame-based methods commonly integrate events into dense representations and adapt these representations to CNNs for further processing. Benefiting from prior knowledge contained in pretrained CNNs, these approaches achieve the highest performance on multiple vision tasks, e.g., event-based recognition [2]- [6], video reconstruction [7], [24], [25], optical flow estimation [26]- [28] . Nevertheless, they usually sacrifice the sparsity of event data, leading to redundant computation and high model complexity [13], [29], thereby limiting eventbased applications on mobile devices, edge computing, etc.…”

Section: Related Workmentioning

confidence: 99%

A Dynamic Graph CNN with Cross-Representation Distillation for Event-Based Recognition

Deng¹,

Chen²,

Xie³

et al. 2023

Preprint

View full text Add to dashboard Cite

It is a popular solution to convert events into dense frame-based representations to use the well-pretrained CNNs in hand. Although with appealing performance, this line of work sacrifices the sparsity/temporal precision of events and usually necessitates heavy-weight models, thereby largely weakening the advantages and real-life application potential of event cameras. A more application-friendly way is to design deep graph models for learning sparse point-based representations from events. Yet, the efficacy of these graph models is far behind the framebased counterpart with two key limitations: (i) simple graph construction strategies without carefully integrating the variant attributes (i.e., semantics, spatial and temporal coordinates) for each vertex, leading to biased graph representation; (ii) deficient learning because the lack of well pretraining models available. Here we solve the first problem by introducing a new event-based graph CNN (EDGCN), with a dynamic aggregation module to integrate all attributes of vertices adaptively. To alleviate the learning difficulty, we propose to leverage the dense representation counterpart of events as a cross-representation auxiliary to supply additional supervision and prior knowledge for the event graph. To this end, we form a frame-to-graph transfer learning framework with a customized hybrid distillation loss to well respect the varying cross-representation gaps across layers. Extensive experiments on multiple vision tasks validate the effectiveness and high generalization ability of our proposed model and distillation strategy 1 .

show abstract

Section: Related Workmentioning

confidence: 99%

A Dynamic Graph CNN with Cross-Representation Distillation for Event-Based Recognition

Deng¹,

Chen²,

Xie³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The capacity of neural networks is surely crucial for their success, but earlier directly-trained SNNs mainly suffer from severe accuracy degradation and are limited to shallow structures and simple tasks. Inspired by the representation power of deep ANNs, more attention has been paid to the design of SNN-oriented network structures, and emerging works such as threshold-dependent batch normalization [62], [243], spiking residual learning [63], [64], attentionbased SNNs [65], [244], and spiking transformer [66], [245] are gradually reducing the performance gap compared to ANNs and demonstrating the great potential of large-scale SNNs for more complicated tasks.…”

Section: B Learning Algorithmsmentioning

confidence: 99%

“…Drawing on research in neuroscience is one of the main lines, such as modeling of spiking neurons [69], [262], learning inspired by synaptic plasticity [263], [264], etc. Another important path is to draw nutrients from the development of traditional AI, including the design of SNNs with increasing network [66] Direct Training Spiking Transformer ImageNet 74.81% Wu(2019) [67] Direct Training CNN DVS-CIFAR10 60.50% Zheng(2021) [62] Direct Training ResNet DVS-CIFAR10 67.80% Yao(2021) [68] Direct Training CNN DVS-CIFAR10 72.00% Fang(2021) [69] Direct Training ResNet DVS-CIFAR10 74.80% Meng(2022) [70] Direct Training ResNet DVS-CIFAR10 78.50% Zhou(2022) [66] Direct Training Spiking Transformer DVS-CIFAR10 80.90% He(2020) [71] Direct Training CNN DVS-Gesture 93.40% Shrestha(2018) [72] Direct Training CNN DVS-Gesture 93.64% Zheng(2021) [62] Direct Training ResNet DVS-Gesture 96.87% Fang(2021) [63] Direct Training ResNet DVS-Gesture 97.92% Yao(2022) [65] Direct Training ResNet DVS-Gesture 98.23% Zhou(2022) [66] Direct Training Spiking Transformer DVS-Gesture 98.30%…”

Section: Key Considerationsmentioning

confidence: 99%

Brain Inspired Computing: A Systematic Survey and Future Trends

Li¹,

Deng²,

Tang³

et al. 2023

Preprint

View full text Add to dashboard Cite

<p>Brain Inspired Computing (BIC) is an emerging research field that aims to build fundamental theories, models, hardware architectures, and application systems toward more general Artificial Intelligence (AI) by learning from the information processing mechanisms or structures/functions of biological nervous systems. It is regarded as one of the most promising research directions for future intelligent computing in the post-Moore era. In the past few years, various new schemes in this field have sprung up to explore more general AI. These works are quite divergent in the aspects of modeling/algorithm, software tool, hardware platform, and benchmark data, since BIC is an interdisciplinary field that consists of many different domains, including computational neuroscience, artificial intelligence, computer science, statistical physics, material science, microelectronics and so forth. This situation greatly impedes researchers from obtaining a clear picture and getting started in the right way. Hence, there is an urgent requirement to do a comprehensive survey in this field to help correctly recognize and analyze such bewildering methodologies. What are the key issues to enhance the development of BIC? What roles do the current mainstream technologies play in the general framework of BIC? Which techniques are truly useful in real-world applications? These questions largely remain open. To address the above issues, in this survey we first clarify the biggest challenge of BIC: how can AI models benefit from the recent advancements in computational neuroscience? With this challenge in mind, we will focus on discussing the concept of BIC and summarize four components of BIC infrastructure development: 1) modeling/algorithm; 2) hardware platform; 3) software tool; and 4) benchmark data. For each component, we will summarize its recent progress, main challenges to resolve, and future trends. On the basis of these studies, we present a general framework for the real-world applications of BIC systems, which is promising to benefit both AI and brain science. Finally, we claim that it is extremely important to build a research ecology to promote prosperity continuously in this field.</p>

show abstract

“…where equation ( 1) is similar to the function of the recurrent layer. The membrane time constant controls the balance between remembering and forgetting −1 (Zhu, Wang, Chang, Li, Huang and Tian (2022)). Thus, it can be considered a simple version of the recurrent layer.…”

Section: Bio-inspired Srnn Modelmentioning

confidence: 99%

SpikeSEE: An Energy-Efficient Dynamic Scenes Processing Framework for Retinal Prostheses

Fang¹,

Zou²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

Intelligent and low-power retinal prostheses are highly demanded in this era, where wearable and implantable devices are used for numerous healthcare applications. In this paper, we propose an energy-efficient dynamic scenes processing framework (SpikeSEE) that combines a spike representation encoding technique and a bio-inspired spiking recurrent neural network (SRNN) model to achieve intelligent processing and extreme low-power computation for retinal prostheses. The spike representation encoding technique could interpret dynamic scenes with sparse spike trains, decreasing the data volume. The SRNN model, inspired by the human retina's special structure and spike processing method, is adopted to predict the response of ganglion cells to dynamic scenes. Experimental results show that the Pearson correlation coefficient of the proposed SRNN model achieves 0.93, which outperforms the state-of-the-art processing framework for retinal prostheses. Thanks to the spike representation and SRNN processing, the model can extract visual features in a multiplication-free fashion. The framework achieves 12 times power reduction compared with the convolutional recurrent neural network (CRNN) processing-based framework. Our proposed SpikeSEE predicts the response of ganglion cells more accurately with lower energy consumption, which alleviates the precision and power issues of retinal prostheses and provides a potential solution for wearable or implantable prostheses.

show abstract

Event-based Video Reconstruction via Potential-assisted Spiking Neural Network

Cited by 75 publications

References 51 publications

A Dynamic Graph CNN with Cross-Representation Distillation for Event-Based Recognition

A Dynamic Graph CNN with Cross-Representation Distillation for Event-Based Recognition

Brain Inspired Computing: A Systematic Survey and Future Trends

SpikeSEE: An Energy-Efficient Dynamic Scenes Processing Framework for Retinal Prostheses

Contact Info

Product

Resources

About