Perception and sensing for autonomous vehicles under adverse weather conditions: A survey

Zhang, Yuxiao; Carballo, Alexander; Yang, Hanting; Takeda, Kazuya

doi:10.1016/j.isprsjprs.2022.12.021

Cited by 182 publications

(73 citation statements)

References 103 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further, these models may need to demonstrate comprehension of the mental state, intentions, and more of humans in the case of interactions or related cases (see [ 2 ] for more details). As demonstrated in [ 3 , 4 , 5 , 6 , 7 ], additional relevant aspects regarding challenges include topics such as perception, navigation, object detection, etc. Each of these factors requires a system to convert sensor data into actions.…”

Section: Related Workmentioning

confidence: 99%

“…These include cases of models trained to predict or identify the source of a driving failure given driving-related data such as speed, steering angle, etc. [ 7 , 9 , 10 , 11 , 12 , 13 ]. These works show an ability to use observations to identify or predict likely failures.…”

Section: Related Workmentioning

confidence: 99%

“…A further relevant topic is the case of route planning in autonomous driving and the potential impact of context on outcome. As seen in [ 6 , 7 , 14 , 15 , 16 ], the success rates for perception can be adversely affected by weather, sensor type, and related factors. Something as commonplace as snow and ice could result in a significant impact on camera performance [ 17 ].…”

Section: Related Workmentioning

confidence: 99%

“…Specific to the context of autonomous systems, we should also consider the issue of perturbation and related attacks on vision and sensing. As seen in [ 7 , 18 , 19 , 20 , 21 , 22 , 23 ], small (often imperceptible) changes to the appearance of an image result in significant changes in classification. As in [ 18 ], a small sticker on a stop sign could prevent detection.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Compensating for Sensing Failures via Delegation in Human–AI Hybrid Systems

Fuchs

Passarella

Conti

2023

Sensors

View full text Add to dashboard Cite

Given the increasing prevalence of intelligent systems capable of autonomous actions or augmenting human activities, it is important to consider scenarios in which the human, autonomous system, or both can exhibit failures as a result of one of several contributing factors (e.g., perception). Failures for either humans or autonomous agents can lead to simply a reduced performance level, or a failure can lead to something as severe as injury or death. For our topic, we consider the hybrid human–AI teaming case where a managing agent is tasked with identifying when to perform a delegated assignment and whether the human or autonomous system should gain control. In this context, the manager will estimate its best action based on the likelihood of either (human, autonomous) agent’s failure as a result of their sensing capabilities and possible deficiencies. We model how the environmental context can contribute to, or exacerbate, these sensing deficiencies. These contexts provide cases where the manager must learn to identify agents with capabilities that are suitable for decision-making. As such, we demonstrate how a reinforcement learning manager can correct the context–delegation association and assist the hybrid team of agents in outperforming the behavior of any agent working in isolation.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Compensating for Sensing Failures via Delegation in Human–AI Hybrid Systems

Fuchs

Passarella

Conti

2023

Sensors

View full text Add to dashboard Cite

show abstract

“…Autonomous vehicles are highly dependent on their perception systems to sense the surrounding environment and detect other road users, as this information is essential for their planning and control systems. However, when adverse weather occurs, retrieved images with low contrast and poor visibility can degrade the performance of the visual algorithms used in autonomous vehicle perception systems, such as detection, tracking, and intention estimation [ 1 , 2 , 3 , 4 , 5 ]. This degradation is caused by tiny particles in the atmosphere that absorb and reflect light [ 6 ].…”

Section: Introductionmentioning

confidence: 99%

Framework for Generation and Removal of Multiple Types of Adverse Weather from Driving Scene Images

Yang

Carballo

Zhang

et al. 2023

Sensors

Self Cite

View full text Add to dashboard Cite

Weather variation in the distribution of image data can cause a decline in the performance of existing visual algorithms during evaluation. Adding additional samples of target domain to training data or using pre-trained image restoration methods such as de-hazing, de-raining, and de-snowing, to improve the quality of input images are two promising solutions. In this work, we propose Multiple Weather Translation GAN (MWTG), a CycleGAN-based, dual-purpose framework that simultaneously learns weather generation and its removal from image data. MWTG consists of four GANs constrained using cycle consistency that carry out domain translation tasks between hazy, rainy, snowy, and clear weather, using an asymmetric approach. To increase network capacity, we employ a spatial feature transform (SFT) layer to fuse the features extracted from the weather layer, which contains high-level domain information from the previous generators. Further, we collect an unpaired, real-world driving dataset recorded under various weather conditions called Realistic Driving Scenes under Bad Weather (RDSBW). We qualitatively and quantitatively evaluate MWTG using the RDSBW and the variation of Cityscapes that synthesize weather effects, eg., FoggyCityscape. Our experimental results suggest that MWTG can generate realistic weather in clear images and also accurately remove noise from weather images. Furthermore, the SOTA pedestrian detector ASCP is shown to achieve an impressive gain in detection precision after image restoration using the proposed MWTG method.

show abstract