Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather

Bijelic, Mario; Gruber, Tobias; Mannan, Fahim; Kraus, Florian; Ritter, Walter; Dietmayer, Klaus; Heide, Felix

doi:10.1109/cvpr42600.2020.01170

Cited by 340 publications

(290 citation statements)

References 60 publications

Supporting

Mentioning

242

Contrasting

Order By: Relevance

“…A more in-depth discussion about millimeter-wave radar and camera sensor coordinate transformations can be found in [ 150 ]. To this end, many deep learning-based fusion algorithms using vision and radar data that are projected onto various domains are reported in the literature [ 47 , 48 , 55 , 56 , 57 , 100 , 151 , 152 , 153 , 154 , 155 , 156 ].…”

Section: Deep Learning-based Multi-sensor Fusion Of Radar and Camementioning

confidence: 99%

“…Mario Bijelic et al [ 156 ] developed a multimodal adverse weather dataset incorporating a camera, Lidar, radar, gated near-infrared (NIR), and far-infrared (FIR) sensory data to detect adverse weather objects for autonomous driving applications. Moreover, they proposed a novel real-time multimodal deep fusion pipeline that exploited the measurement entropy that adaptively fused the multiple sensory data, thus avoiding the proposal-based level fusion methods.…”

Section: Deep Learning-based Multi-sensor Fusion Of Radar and Camementioning

confidence: 99%

“…Moreover, M. Bijelic [ 156 ] proposed an adaptive deep fusion scheme utilizing Lidar, RGB camera, gated camera, and radar sensory data. All the multi-sensor data were projected onto a common camera plane.…”

Section: Deep Learning-based Multi-sensor Fusion Of Radar and Camementioning

confidence: 99%

“…For example, several studies performed radar and vision deep learning-based fusion when building their architectures on top of one-stage object detector [ 53 , 54 , 57 , 58 , 154 , 159 ] or two-stage object detector models [ 47 , 153 , 156 ]. However, some studies extended the network structure to capture the peculiarities and more semantic information in radar signals [ 55 ].…”

Section: Deep Learning-based Multi-sensor Fusion Of Radar and Camementioning

confidence: 99%

See 3 more Smart Citations

Application of Deep Learning on Millimeter-Wave Radar Signals: A Review

Abdu

Zhang

et al. 2021

Sensors

View full text Add to dashboard Cite

The progress brought by the deep learning technology over the last decade has inspired many research domains, such as radar signal processing, speech and audio recognition, etc., to apply it to their respective problems. Most of the prominent deep learning models exploit data representations acquired with either Lidar or camera sensors, leaving automotive radars rarely used. This is despite the vital potential of radars in adverse weather conditions, as well as their ability to simultaneously measure an object’s range and radial velocity seamlessly. As radar signals have not been exploited very much so far, there is a lack of available benchmark data. However, recently, there has been a lot of interest in applying radar data as input to various deep learning algorithms, as more datasets are being provided. To this end, this paper presents a survey of various deep learning approaches processing radar signals to accomplish some significant tasks in an autonomous driving application, such as detection and classification. We have itemized the review based on different radar signal representations, as it is one of the critical aspects while using radar data with deep learning models. Furthermore, we give an extensive review of the recent deep learning-based multi-sensor fusion models exploiting radar signals and camera images for object detection tasks. We then provide a summary of the available datasets containing radar data. Finally, we discuss the gaps and important innovations in the reviewed papers and highlight some possible future research prospects.

show abstract

Section: Deep Learning-based Multi-sensor Fusion Of Radar and Camementioning

confidence: 99%

Section: Deep Learning-based Multi-sensor Fusion Of Radar and Camementioning

confidence: 99%

Section: Deep Learning-based Multi-sensor Fusion Of Radar and Camementioning

confidence: 99%

Section: Deep Learning-based Multi-sensor Fusion Of Radar and Camementioning

confidence: 99%

See 2 more Smart Citations

Application of Deep Learning on Millimeter-Wave Radar Signals: A Review

Abdu

Zhang

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Finally, the SeeingThroughFog dataset [33] was developed in the context of the DENSE project. It records 10,000 km of driving in Northern Europe under different weather and illumination conditions in February and December 2019.…”

Section: Existing Fog Detection Datasetsmentioning

confidence: 99%

Sensors on the Move: Onboard Camera-Based Real-Time Traffic Alerts Paving the Way for Cooperative Roads

Iparraguirre

Amundarain

Brazalez

et al. 2021

Sensors

View full text Add to dashboard Cite

European road safety has improved greatly in recent decades. However, the current numbers are still far away to reach the European Commission’s road safety targets. In this context, Cooperative Intelligent Transport Systems (C-ITS) are expected to significantly improve road safety, traffic efficiency and comfort of driving, by helping the driver to make better decisions and adapt to the traffic situation. This paper puts forward two vision-based applications for traffic sign recognition (TSR) and real-time weather alerts, such as for fog-banks. These modules will support operators in road infrastructure maintenance tasks as well as drivers, giving them valuable information via C-ITS messages. Different state-of-the-art methods are analysed using both publicly available datasets (GTSB) as well as our own image databases (Ceit-TSR and Ceit-Foggy). The selected models for TSR implementation are based on Aggregated Chanel Features (ACF) and Convolutional Neural Networks (CNN) that reach more than 90% accuracy in real time. Regarding fog detection, an image feature extraction method on different colour spaces is proposed to differentiate sunny, cloudy and foggy scenes, as well as its visibility level. Both applications are already running in an onboard probe vehicle system.

show abstract

P‐16.2: A Review of Obstacle Detection based on Multi‐sensor Fusion

Dai

2022

Symp Digest of Tech Papers

View full text Add to dashboard Cite

According to the classification of using multiple sensors, the obstacle detection and recognition technology of unmanned driving is summarized, and the main technical means, algorithms, advantages and disadvantages, difficulties and so on are discussed. These methods mainly include lidar and camera fusion, camera and radar fusion and lidar, camera and radar fusion. In addition, some research achievements in this field in recent years are introduced, and some fusion technologies used in this field are briefly analyzed and summarized.

show abstract

Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather

Cited by 340 publications

References 60 publications

Application of Deep Learning on Millimeter-Wave Radar Signals: A Review

Application of Deep Learning on Millimeter-Wave Radar Signals: A Review

Sensors on the Move: Onboard Camera-Based Real-Time Traffic Alerts Paving the Way for Cooperative Roads

P‐16.2: A Review of Obstacle Detection based on Multi‐sensor Fusion

Contact Info

Product

Resources

About