Fusing Event-based and RGB camera for Robust Object Detection in Adverse Conditions

Tomy, Abhishek; Paigwar, Anshul; Mann, Khushdeep Singh; Renzaglia, Alessandro; Laugier, Christian

doi:10.1109/icra46639.2022.9812059

Cited by 44 publications

(29 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of the investigations focused on the fusion of events and frames, thanks to sensors offering both modalities like the DAVIS camera [2]. These works include frame interpolation and deblurring [26,27,31], feature tracking [8,20], object detection [3,16,38], or even steering prediction [13].…”

Section: Fusion Of Events and Other Modalitiesmentioning

confidence: 99%

Learning to Estimate Two Dense Depths from LiDAR and Event Data

Brebion¹,

Moreau²,

Davoine³

2023

Preprint

View full text Add to dashboard Cite

Event cameras do not produce images, but rather a continuous flow of events, which encode changes of illumination for each pixel independently and asynchronously. While they output temporally rich information, they lack any depth information which could facilitate their use with other sensors. LiDARs can provide this depth information, but are by nature very sparse, which makes the depth-to-event association more complex. Furthermore, as events represent changes of illumination, they might also represent changes of depth; associating them with a single depth is therefore inadequate. In this work, we propose to address these issues by fusing information from an event camera and a LiDAR using a learning-based approach to estimate accurate dense depth maps. To solve the "potential change of depth" problem, we propose here to estimate two depth maps at each step: one "before" the events happen, and one "after" the events happen. We further propose to use this pair of depths to compute a depth difference for each event, to give them more context. We train and evaluate our network, ALED, on both synthetic and real driving sequences, and show that it is able to predict dense depths with an error reduction of up to 61% compared to the current state of the art. We also demonstrate the quality of our 2-depths-toevent association, and the usefulness of the depth difference information. Finally, we release SLED, a novel synthetic dataset comprising events, LiDAR point clouds, RGB images, and dense depth maps.

show abstract

Section: Fusion Of Events and Other Modalitiesmentioning

confidence: 99%

Learning to Estimate Two Dense Depths from LiDAR and Event Data

Brebion¹,

Moreau²,

Davoine³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Thus the temporal approaches would incorporate meaningful input history, instead of treating each split, of the input stream, independently, yet at the expense of higher computation. Although the single-modal event-based approaches are promising, their performance typically lags both the frame-based solutions, under normal conditions, as well as the combined solutions, which incorporate both modalities 8 , 29 , 33 – 35 The main reason behind this is due to the nature of the event data, especially in scenes where there is limited motion.…”

Section: Related Workmentioning

confidence: 99%

“…Although the single-modal event-based approaches are promising, their performance typically lags both the frame-based solutions, under normal conditions, as well as the combined solutions, which incorporate both modalities. 8,29,[33][34][35] The main reason behind this is due to the nature of the event data, especially in scenes where there is limited motion.…”

Section: Event-based Approachesmentioning

confidence: 99%

High-temporal-resolution event-based vehicle detection and tracking

Shair

Rawashdeh

2022

Opt. Eng.

View full text Add to dashboard Cite

Event-based vision has been rapidly growing in recent years justified by the unique characteristics, such as its high temporal resolutions (∼1 μs), high dynamic range (>120 dB), and output latency of only a few microseconds. Our work further explores a hybrid, multimodal approach for object detection and tracking that leverages state-of-the-art frame-based detectors complemented by hand-crafted event-based methods to improve the overall tracking performance with minimal computational overhead. The methods presented include event-based bounding box (BB) refinement that improves the precision of the resulting BBs, as well as a continuous event-based object detection method, to recover missed detections and generate interframe detections that enable a high-temporal-resolution tracking output. The advantages of these methods are quantitatively verified by an ablation study using the higher order tracking accuracy (HOTA) metric. Results show significant performance gains resembled by an improvement in the HOTA from 56.6%, using only frames, to 64.1% and 64.9%, for the event and edgebased mask configurations combined with the two methods proposed, at the baseline frame rate of 24 Hz. Likewise, incorporating these methods with the same configurations has improved HOTA from 52.5% to 63.1% and from 51.3% to 60.2% at the high-temporal-resolution tracking rate of 384 Hz. Finally, a validation experiment is conducted to analyze the real-world singleobject tracking performance using high-speed LiDAR. Empirical evidence shows that our approaches provide significant advantages compared to using frame-based object detectors at the baseline frame rate of 24 Hz and higher tracking rates of up to 500 Hz.

show abstract

“…In reality, the RGB-based detection fails to enable robust perception under image corruptions or extreme weather conditions. Meanwhile, auxiliary sensors, such as LiDARs, are extremely bulky and costly [142]. Therefore, event-based detectors are introduced to overcome the dilemma, especially in challenging visual conditions [133].…”

Section: Object Detection and Trackingmentioning

confidence: 99%

“…The third category of attempts combines the advantages of event images and RGB images [135], [136], [145], [146]. Tomy et al [137] proposed a representative framework that fuses the information from the event-and frame-based cameras for better detection accuracy In such cases, introducing event cameras to address this problem is of great value.…”

Section: Object Detection and Trackingmentioning

confidence: 99%

Pan‐risk factor for a comprehensive cardiovascular health management

Zheng

et al. 2022

Journal of Diabetes

View full text Add to dashboard Cite

Cardiovascular diseases (CVDs) have become the leading cause of death in China. CVDs are mainly caused by multiple well‐known modifiable risk factors that are affected by socioeconomic and environmental determinants, lifestyle and behavioral choices, and familial and genetic predispositions. With more risk factors proved to be associated with CVD occurrence, the concept “pan‐risk factor” is proposed in this review to indicate all discovered and yet‐to‐be‐discovered CVD risk factors for comprehensive primary prevention of CVD. Recognizing more factors and their roles in CVD development and progression is the first step in reducing the ever‐increasing burden of CVD. This review is an overview of the pan‐risk factor whose associations with CVD outcomes have been established. Along with the accumulation of scientific evidence, an increasing number of risk factors will be discovered and included in the list of pan‐risk factors.

show abstract

Fusing Event-based and RGB camera for Robust Object Detection in Adverse Conditions

Cited by 44 publications

References 27 publications

Learning to Estimate Two Dense Depths from LiDAR and Event Data

Learning to Estimate Two Dense Depths from LiDAR and Event Data

High-temporal-resolution event-based vehicle detection and tracking

Pan‐risk factor for a comprehensive cardiovascular health management

Contact Info

Product

Resources

About