Event-Based Pedestrian Detection Using Dynamic Vision Sensors

Abstract

Pedestrian detection has attracted great research attention in video surveillance, traffic statistics, and especially in autonomous driving. To date, almost all pedestrian detection solutions are derived from conventional framed-based image sensors with limited reaction speed and high data redundancy. Dynamic vision sensor (DVS), which is inspired by biological retinas, efficiently captures the visual information with sparse, asynchronous events rather than dense, synchronous frames. It can eliminate redundant data transmission and avoid motion blur or data leakage in high-speed imaging applications. However, it is usually impractical to directly apply the event streams to conventional object detection algorithms. For this issue, we first propose a novel event-to-frame conversion method by integrating the inherent characteristics of events more efficiently. Moreover, we design an improved feature extraction network that can reuse intermediate features to further reduce the computational effort. We evaluate the performance of our proposed method on a custom dataset containing multiple real-world pedestrian scenes. The results indicate that our proposed method raised its pedestrian detection accuracy by about 5.6–10.8%, and its detection speed is nearly 20% faster than previously reported methods. Furthermore, it can achieve a processing speed of about 26 FPS and an AP of 87.43% when implanted on a single CPU so that it fully meets the requirement of real-time detection.

Real-Time Event-Based Unsupervised Feature Consolidation and Tracking for Space Situational Awareness

“…Similar activity, saliency and background noise filters are common tracker components in neuromorphic literature. Examples include band-pass filtering in the frequency domain (Scheerlinck et al, 2019), Leaky Integrate and Fire (LIF) neuron based filters of Surface of Activated Events (SAE) with frequency thresholding (Wan et al, 2021), spatiotemporal neighborhood filtering by examining the density of surrounding events (Feng et al, 2020) and biologically inspired multi-layered receptive fields for filtering and compression (Barrios-Avilés et al, 2018). While these methods each present many advantages, the focus of FIESTA is to reduce processing time and requires highly simplified filters.…”

Unconventional Visual Sensors for Autonomous Vehicles

“…Hu et al [179], [180] illustrate proposed grafted networks and events synthesis from video frames with a car detection use case. Pedestrian detection is also important and is explored in [181]- [184], in which individual datasets are utilized according to specific cameras. Chen et al [181] compare different accumulation methods coupled with early fusion and late fusion schemes.…”

“…Cladera et al [183] implement a BNN (binary neural networks) on FPGA for fast detection. Wan et al [184] propose a Pedestrian-SARI dataset and alternative events representations for asynchronous CNN detection. Lane extraction problem is investigated in Cheng et al [177], in which a DET dataset, labeled lane markings in HD event camera, is released to be public.…”

Event-based YOLO object detection: proof of concept for forward perception system

“…This mixed approach was evaluated with the best average precision of 92% using YOLOv3. Jixiang Wan et al [18], proposed a novel event-to-frame-based approach further to detect pedestrian scenes. The network performed with an average precision of 81.43% at 26 frames per second (FPS).…”