Guided Event Filtering: Synergy between Intensity Images and Neuromorphic Events for High Performance Imaging

Duan, Peiqi; Wang, Zihao; Shi, Boxin; Cossairt, Oliver; Huang, Tiejun; Katsaggelos, Aggelos K.

doi:10.1109/tpami.2021.3113344

Cited by 17 publications

(8 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DAVIS 346 camera was mounted on top of a table and shot a monitor playing the need-forspeed (NFS) [38] dataset. RGB DAVIS [23] provides 20 real event sequences from a DAVIS 240 camera, including indoor and outdoor scenes, as well as high-resolution frames from a conventional RGB camera. Although these datasets provide a large quantity of realistic noisy data, they were collected under limited lighting conditions; some of them (e.g.DVSNOISE20 and ENFS) contain only restricted motion, which cannot cover authentic camera working scenarios.…”

Section: A Event Denoising Datasetsmentioning

confidence: 99%

“…EV-Gait [36] performs local plane optical flow estimation and filters noisy events to smooth the optical flow surface. Afterward, the guided event filter (GEF) [23] combines the gradient of active pixel sensor (APS) frames. In contrast, time surface (TS) [30], [34], [35] transforms events from unit impulses into a representation that is monotonically decreasing with time, which solves the sparsity problem in the local plan fitting process.…”

Section: Event-based Denoising Algorithmsmentioning

confidence: 99%

“…• GEF [23] provides two denoising modes. In the frameguide mode, the guided event filter (GEF) extracts mutual structures between the event frame (project along optical flow) and the gradient of the APS image (by Sobel operator), then deletes unreasonable events and reallocates back to spatiotemporal space.…”

Section: A Event Denoising Baselinesmentioning

confidence: 99%

See 2 more Smart Citations

E-MLB: Multilevel Benchmark for Event-Based Camera Denoising

Ding

Chen

Wang

et al. 2024

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Event cameras, such as dynamic vision sensors (DVS), are biologically inspired vision sensors that have advanced over conventional cameras in high dynamic range, low latency and low power consumption, showing great application potential in many fields. Event cameras are more sensitive to junction leakage current and photocurrent as they output differential signals, losing the smoothing function of the integral imaging process in the RGB camera. The logarithmic conversion further amplifies noise, especially in low-contrast conditions. Recently, researchers proposed a series of datasets and evaluation metrics but limitations remain: 1) the existing datasets are small in scale and insufficient in noise diversity, which cannot reflect the authentic working environments of event cameras; and 2) the existing denoising evaluation metrics are mostly referenced evaluation metrics, relying on APS information or manual annotation. To address the above issues, we construct a largescale event denoising dataset (multilevel benchmark for event denoising, E-MLB) for the first time, which consists of 100 scenes, each with four noise levels, that is 12 times larger than the largest existing denoising dataset. We also propose the first nonreference event denoising metric, the event structural ratio (ESR), which measures the structural intensity of given events. ESR is inspired by the contrast metric, but is independent of the number of events and projection direction. Based on the proposed benchmark and ESR, we evaluate the most representative denoising algorithms, including classic and SOTA, and provide denoising baselines under various scenes and noise levels. The corresponding results and codes are available at https://github.com/KugaMaxx/cuke-emlb.

show abstract

Section: A Event Denoising Datasetsmentioning

confidence: 99%

Section: Event-based Denoising Algorithmsmentioning

confidence: 99%

See 1 more Smart Citation

E-MLB: Multilevel Benchmark for Event-Based Camera Denoising

Ding

Chen

Wang

et al. 2024

IEEE Trans. Multimedia

View full text Add to dashboard Cite

show abstract

“…In this case, many algorithms and applications that require alignment of events and images cannot be implemented, e.g.,. 11,53 We first obtain the disparity map using the proposed multi-modal stereo method. Each value in the disparity map indicates the number of pixels that need to be shifted horizontally.…”

Section: Connecting Events and Intensitymentioning

confidence: 99%

Self-Supervised Intensity-Event Stereo Matching

Gu¹,

Zhou²,

Wo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Event cameras are novel bio-inspired vision sensors that output pixel-level intensity changes in microsecond accuracy with a high dynamic range and low power consumption. Despite these advantages, event cameras cannot be directly applied to computational imaging tasks due to the inability to obtain high-quality intensity and events simultaneously. This paper aims to connect a standalone event camera and a modern intensity camera so that the applications can take advantage of both two sensors. We establish this connection through a multi-modal stereo matching task. We first convert events to a reconstructed image and extend the existing stereo networks to this multi-modality condition. We propose a self-supervised method to train the multi-modal stereo network without using ground truth disparity data. The structure loss calculated on image gradients is used to enable self-supervised learning on such multi-modal data. Exploiting the internal stereo constraint between views with different modalities, we introduce general stereo loss functions, including disparity cross-consistency loss and internal disparity loss, leading to improved performance and robustness compared to existing approaches. The experiments demonstrate the effectiveness of the proposed method, especially the proposed general stereo loss functions, on both synthetic and real datasets. At last, we shed light on employing the aligned events and intensity images in downstream tasks, e.g., video interpolation application.

show abstract

“…However, new algorithms are needed to deal with the unconventional type of data they produce (per-pixel asynchronous brightness changes, called events) and unlock their advantages [ 4 ]. Contrast maximization (CMax) is an event processing framework that provides state-of-the-art results on several tasks, such as rotational motion estimation [ 5 , 6 ], feature flow estimation and tracking [ 7 , 8 , 9 , 10 , 11 ], ego-motion estimation [ 12 , 13 , 14 ], 3D reconstruction [ 12 , 15 ], optical flow estimation [ 16 , 17 , 18 , 19 ], motion segmentation [ 20 , 21 , 22 , 23 , 24 ], guided filtering [ 25 ], and image reconstruction [ 26 ].…”

Section: Introductionmentioning

confidence: 99%

Event Collapse in Contrast Maximization Frameworks

Shiba

Aoki

Gallego

2022

Sensors

View full text Add to dashboard Cite

Contrast maximization (CMax) is a framework that provides state-of-the-art results on several event-based computer vision tasks, such as ego-motion or optical flow estimation. However, it may suffer from a problem called event collapse, which is an undesired solution where events are warped into too few pixels. As prior works have largely ignored the issue or proposed workarounds, it is imperative to analyze this phenomenon in detail. Our work demonstrates event collapse in its simplest form and proposes collapse metrics by using first principles of space–time deformation based on differential geometry and physics. We experimentally show on publicly available datasets that the proposed metrics mitigate event collapse and do not harm well-posed warps. To the best of our knowledge, regularizers based on the proposed metrics are the only effective solution against event collapse in the experimental settings considered, compared with other methods. We hope that this work inspires further research to tackle more complex warp models.

show abstract

Guided Event Filtering: Synergy between Intensity Images and Neuromorphic Events for High Performance Imaging

Cited by 17 publications

References 60 publications

E-MLB: Multilevel Benchmark for Event-Based Camera Denoising

E-MLB: Multilevel Benchmark for Event-Based Camera Denoising

Self-Supervised Intensity-Event Stereo Matching

Event Collapse in Contrast Maximization Frameworks

Contact Info

Product

Resources

About