Object Detection with Spiking Neural Networks on Automotive Event Data

Cordone, Loïc; Miramond, Benoît; Thierion, Philippe

doi:10.1109/ijcnn55064.2022.9892618

Cited by 61 publications

(34 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…So far, SNNs have been used for classification tasks like image recognition [13], [29], object detection [30], [31], or motion segmentation [32]. Only a few works employed them for regression tasks.…”

Section: Related Workmentioning

confidence: 99%

StereoSpike: Depth Learning With a Spiking Neural Network

Rançon¹,

Cuadrado-Anibarro²,

Cottereau³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Depth estimation is an important computer vision task, useful in particular for navigation in autonomous vehicles, or for object manipulation in robotics. Here, we propose to solve it using StereoSpike, an end-to-end neuromorphic approach, combining two event-based cameras and a Spiking Neural Network (SNN) with a modified U-Net-like encoder-decoder architecture. More specifically, we used the Multi Vehicle Stereo Event Camera Dataset (MVSEC). It provides a depth ground-truth, which was used to train StereoSpike in a supervised manner, using surrogate gradient descent. We propose a novel readout paradigm to obtain a dense analog prediction -the depth of each pixel-from the spikes of the decoder. We demonstrate that this architecture generalizes very well, even better than its non-spiking counterparts, leading to near state-of-the-art test accuracy. To the best of our knowledge, it is the first time that such a large-scale regression problem is solved by a fully spiking neural network. Finally, we show that very low firing rates (<5%) can be obtained via regularization, with a minimal cost in accuracy. This means that StereoSpike could be efficiently implemented on neuromorphic chips, opening the door for low power and real time embedded systems.

show abstract

Section: Related Workmentioning

confidence: 99%

StereoSpike: Depth Learning With a Spiking Neural Network

Rançon¹,

Cuadrado-Anibarro²,

Cottereau³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Mapping the event stream to a dense representation [24], [25], [26], [27], [28], [29] is an important approach. Even if these methods lose some of the event's temporal resolution, they gain in terms of accuracy and scalability.…”

Section: A Feature Representationmentioning

confidence: 99%

“…A histogram method called ''event cube'' [24] has been proposed to combine the simplicity of histograms with the temporal information of time surfaces. Voxel grids [25], [26], voxel cube [27], graph [28], and DART (distribution aware retinal transform) [29] are successful examples of dense descriptors, statistical or structural, for event-based data. They preserve partially temporal information and asynchronous characteristics.…”

Section: A Feature Representationmentioning

confidence: 99%

“…Hybrid-SNN [42] proposes a partial solution by presenting an hybrid neural network composed of a SNN backbone for efficient event-based feature extraction, and an ANN head to solve object detection tasks. [27] offers a SNN composed of a spiking backbone and SSD bounding box regression heads to achieve qualitative results. Unfortunately, these SNNs still need to catch up to CNNs in terms of accuracy.…”

Section: ) Snnmentioning

confidence: 99%

See 1 more Smart Citation

Spike-Event Object Detection for Neuromorphic Vision

Wang

2023

IEEE Access

View full text Add to dashboard Cite

Neuromorphic vision is one of the novel research fields that study neuromorphic cameras and spiking neural networks (SNNs) for computer vision. Instead of computing on frame-based images, spike events are streamed from neuromorphic cameras, and novel object detection algorithms have to deal with spike events to achieve detection tasks. In this paper, we propose a solution to the novel object detection method with spike events. Spike events are first encoded to event images according to the computational methodology of neuromorphic theory. The event images can be realized as change-detected images of moving objects with a high frame rate. A redesigned deep learning framework is proposed for object detection to process the event images. We propose a deep SNN method that is achieved by the conversion of successful convolution neural networks but trained by event images. The networks with multiscale representation are discussed and designed in our method. We also design a semi-automatic data labeling method to build event-image datasets by object tracking algorithms. The proposed solution, therefore, includes spike event encoding, a redesigned deep SNN, and an event-image data augmentation algorithm. Experiments are conducted not only on the MNIST-DVS dataset, which is a benchmark dataset for the study of neuromorphic vision but also on our event pedestrian detection dataset. The experimental results show that the performance of the deep SNN trained with our augmented data is close to the model trained on manually labeled data. A performance comparison based on the PAFBenchmark dataset shows that our proposed method has higher accuracy than existing SNN methods, and better energy efficiency and lower energy consumption than existing CNN methods. It demonstrates that our deep SNN method is a feasible solution for the study of neuromorphic vision. The intuition that deep SNN trained with more learning data can achieve better accuracy is also confirmed for this brand-new research field.

show abstract

“…Evaluation of state-of-the-art neuromorphic vision algorithms: We evaluate 13 existing neuromorphic vision algorithms (3 SNNs and 10 DNNs) on our dataset and report on their performance along with insights about the comparative performance. In particular, Experimental evaluation of the dataset proposed in this work has been performed by training Visual Geometry Group (VGG) [52], DenseNet [27], and MobileNet [26] SNNs which have been previously benchmarked in the literature on a large collection of neuromorphic data [12]. The results of the evaluation are competitive to the benchmarking results for each trained SNN.…”

Section: Introductionmentioning

confidence: 99%

NU-AIR -- A Neuromorphic Urban Aerial Dataset for Detection and Localization of Pedestrians and Vehicles

Iaboni¹,

Kelly²,

Abichandani³

2023

Preprint

View full text Add to dashboard Cite

Annotated imagery capturing pedestrians and vehicles in an urban environment can be used to train Neural Networks (NNs) for machine vision tasks. This paper presents the first open-source aerial neuromorphic dataset that captures pedestrians and vehicles moving in an urban environment. The dataset, titled NU-AIR, features 70.75 minutes of event footage acquired with a 640 × 480 resolution neuromorphic sensor mounted on a quadrotor operating in an urban environment. Crowds of pedestrians, different types of vehicles, and street scenes at a busy urban intersection are captured at different elevations and illumination conditions. Manual bounding box annotations of vehicles and pedestrians contained in the recordings are provided at a frequency of 30 Hz, yielding 93,204 labels in total. Evaluation of the dataset's fidelity is performed by training three Spiking Neural Networks (SNNs) and ten Deep Neural Networks (DNNs). The mean average precision (mAP) accuracy results achieved for the testing set evaluations are on-par with results reported for similar SNNs and DNNs on established neuromorphic benchmark datasets. All data and Python code to voxelize the data and subsequently train SNNs/DNNs has been open-sourced.

show abstract

Object Detection with Spiking Neural Networks on Automotive Event Data

Cited by 61 publications

References 17 publications

StereoSpike: Depth Learning With a Spiking Neural Network

StereoSpike: Depth Learning With a Spiking Neural Network

Spike-Event Object Detection for Neuromorphic Vision

NU-AIR -- A Neuromorphic Urban Aerial Dataset for Detection and Localization of Pedestrians and Vehicles

Contact Info

Product

Resources

About