Exploiting Spatial Sparsity for Event Cameras with Visual Transformers

Wang, Zuowen; Hu, Yuhuang; Liu, Shih-Chii

doi:10.1109/icip46576.2022.9897432

Cited by 19 publications

(4 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the recent success of transformer networks in natural language processing, and subsequent use in computer vision, there has been interest in transformer architectures with events. In fact, some characteristics of these visual transformers make them particularly compatible with event data, such as the handling of variable-length inputs and reduced computation by leveraging the event data's ability to highlight active regions [259]. Table 14 presents a summary of recent works in event camera literature that use transformer architectures, along with their applications and the event representation used.…”

Section: E Need Of Transformersmentioning

confidence: 99%

Event Cameras in Automotive Sensing: A Review

Shariff,

Dilmaghani,

Kielty

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Event cameras (EC) represent a paradigm shift and are emerging as valuable tools in the automotive industry, particularly for in-cabin and out-of-cabin monitoring. These cameras capture pixel intensity changes as "events" with ultra-low latency, making them suitable for real-time applications. In the context of in-cabin monitoring, EC offer solution for driver and passenger tracking, enhancing safety and comfort. For out-of-cabin monitoring, they excel in tracking objects and detecting potential hazards on the road. This article explores the applications, benefits, and challenges of event cameras in these two critical domains within the automotive industry. This review also highlights relevant datasets and methodologies, enabling researchers to make informed decisions tailored to their specific vehicular-technology and place their work in the broader context of EC sensing. Through an exploration of the hardware, the complexities of data processing, and customized algorithms for both in-cabin and out-of-cabin surveillance, this paper outlines a framework encompassing methodologies, tools, and datasets critical for the implementation of event camera sensing in automotive systems. INDEX TERMSNeuromorphic Event Camera, Event based Vision, Driver Monitoring System, Out-of-cabin Monitoring, In-Cabin Monitoring.

show abstract

Section: E Need Of Transformersmentioning

confidence: 99%

Event Cameras in Automotive Sensing: A Review

Shariff,

Dilmaghani,

Kielty

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Evaluated on N-MNIST [25] and ASL-DVS [26], table 1 presents the top-1 recognition accuracy on raw events, on raw events with 50% random noise, on the encrypted events by the competitor (Du et al [3]) and on our encrypted events. The approaches, which are either grid-based [13,29,33] or graph-based [32], can still have good performance when the data are filled with random noise, but they fail for recognition on the encrypted ones. In figure 5, we diagnose the learning process and the feature responses of a trained network.…”

Section: Attacks From High-level Neuromorphic Reasoningmentioning

confidence: 99%

Event encryption: rethinking privacy exposure for neuromorphic imaging

Zhang,

Zhu,

Lam

2024

Neuromorph. Comput. Eng.

View full text Add to dashboard Cite

Bio-inspired neuromorphic cameras sense illumination changes on a per-pixel basis and generate spatiotemporal streaming events within microseconds in response, offering visual information with high temporal resolution over a high dynamic range. Such devices often serve in surveillance systems due to their applicability and robustness in environments with high dynamics and harsh lighting, where they can still supply clearer recordings than traditional imaging. In other words, when it comes to privacy-relevant cases, neuromorphic cameras also expose more sensitive data and pose serious security threats. Therefore, asynchronous event streams necessitate careful encryption before transmission and usage. This work discusses several potential attack scenarios and approaches event encryption from the perspective of neuromorphic noise removal, in which we inversely introduce well-crafted noise into raw events until they are obfuscated. Our evaluations show that the encrypted events can effectively protect information from attacks of low-level visual reconstruction and high-level neuromorphic reasoning, and thus feature dependable privacy-preserving competence. The proposed solution gives impetus to the security of event data and paves the way to a highly encrypted technique for privacy-protective neuromorphic imaging.

show abstract

“…In event-based vision, attention-based components have found applications in classification [42,53] and image reconstruction [54], and monocular depth estimation [31], but their use in object detection has yet to be investigated.…”

Section: Vision Transformers For Spatio-temporal Datamentioning

confidence: 99%

Recurrent Vision Transformers for Object Detection with Event Cameras

Gehrig,

Scaramuzza

2023

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

We present Recurrent Vision Transformers (RVTs), a novel backbone for object detection with event cameras. Event cameras provide visual information with submillisecond latency at a high-dynamic range and with strong robustness against motion blur. These unique properties offer great potential for low-latency object detection and tracking in time-critical scenarios. Prior work in event-based vision has achieved outstanding detection performance but at the cost of substantial inference time, typically beyond 40 milliseconds. By revisiting the high-level design of recurrent vision backbones, we reduce inference time by a factor of 6 while retaining similar performance. To achieve this, we explore a multi-stage design that utilizes three key concepts in each stage: First, a convolutional prior that can be regarded as a conditional positional embedding. Second, local and dilated global self-attention for spatial feature interaction. Third, recurrent temporal feature aggregation to minimize latency while retaining temporal information. RVTs can be trained from scratch to reach state-of-the-art performance on event-based object detection -achieving an mAP of 47.2% on the Gen1 automotive dataset. At the same time, RVTs offer fast inference (< 12 ms on a T4 GPU) and favorable parameter efficiency (5× fewer than prior art). Our study brings new insights into effective design choices that can be fruitful for research beyond event-based vision.

show abstract

Exploiting Spatial Sparsity for Event Cameras with Visual Transformers

Cited by 19 publications

References 12 publications

Event Cameras in Automotive Sensing: A Review

Event Cameras in Automotive Sensing: A Review

Event encryption: rethinking privacy exposure for neuromorphic imaging

Recurrent Vision Transformers for Object Detection with Event Cameras

Contact Info

Product

Resources

About