Modeling Perception Errors towards Robust Decision Making in Autonomous Vehicles

Piazzoni, Andrea; Cherian, Jim; Slavík, Martin; Dauwels, Justin

doi:10.24963/ijcai.2020/483

Cited by 16 publications

(20 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Therefore, the training data for the surrogate detector would be D = {s i , ỹi } k i=1 , and although we have defined f as a function of all ground-truth objects in the scene, in this paper our implementation factorises over each agent in the scene and acts on a single agent basis. This results in a surrogate model which by design cannot predict False Positive detections, as is the case in most of the perception error model literature (see [29,30,12,42,21]). Choosing a more general surrogate model would avoid this limitation, however, we consider this out of the scope of the present work.…”

Section: Methodsmentioning

confidence: 99%

A Step Towards Efficient Evaluation of Complex Perception Tasks in Simulation

Sadeghi¹,

Rogers²,

Gunn³

et al. 2021

Preprint

View full text Add to dashboard Cite

There has been increasing interest in characterising the error behaviour of systems which contain deep learning models before deploying them into any safety-critical scenario. However, characterising such behaviour usually requires large-scale testing of the model that can be extremely computationally expensive for complex real-world tasks. For example, tasks involving compute intensive object detectors as one of their components. In this work, we propose an approach that enables efficient large-scale testing using simplified low-fidelity simulators and without the computational cost of executing expensive deep learning models. Our approach relies on designing an efficient surrogate model corresponding to the compute intensive components of the task under test. We demonstrate the efficacy of our methodology by evaluating the performance of an autonomous driving task in the Carla [8] simulator with reduced computational expense by training efficient surrogate models for PIXOR [40] and CenterPoint [41] LiDAR detectors, whilst demonstrating that the accuracy of the simulation is maintained.Preprint. Under review.

show abstract

Section: Methodsmentioning

confidence: 99%

A Step Towards Efficient Evaluation of Complex Perception Tasks in Simulation

Sadeghi¹,

Rogers²,

Gunn³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, Ref. [65] models perception errors while considering the effect they have on robust decision making. In the context of validation of sensor models, Ref.…”

Section: Relevance For Safety-oriented Perception Testingmentioning

confidence: 99%

“…An impact analysis of perceptual errors on the downstream driving function has also already been described in Ref. [65].…”

Section: Uncertainty Forecastingmentioning

confidence: 99%

A Review of Testing Object-Based Environment Perception for Safe Automated Driving

2022

View full text Add to dashboard Cite

Safety assurance of automated driving systems must consider uncertain environment perception. This paper reviews literature addressing how perception testing is realized as part of safety assurance. The paper focuses on testing for verification and validation purposes at the interface between perception and planning, and structures the analysis along the three axes (1) test criteria and metrics, (2) test scenarios, and (3) reference data. Furthermore, the analyzed literature includes related safety standards, safety-independent perception algorithm benchmarking, and sensor modeling. It is found that the realization of safety-oriented perception testing remains an open issue since challenges concerning the three testing axes and their interdependencies currently do not appear to be sufficiently solved.

show abstract

“…Simulation Environment This work presents simulation-based perception testing for autonomous vehicles to evaluate perception object detection systems [1]. This work introduces a comprehensive study to evaluate closed-loop simulation testing for autonomous driving models that include deep learning detection modules learning [1] . LGSVL Road Scenarios [1] In addition, the perception system imports object detection algorithms to simulate the testing scenarios.…”

Section: Experiments Establishmentioning

confidence: 99%

“…This work introduces a comprehensive study to evaluate closed-loop simulation testing for autonomous driving models that include deep learning detection modules learning [1] . LGSVL Road Scenarios [1] In addition, the perception system imports object detection algorithms to simulate the testing scenarios. In the meantime, it involves an evaluation suite to describe scenarios in a virtual environment.…”

Section: Experiments Establishmentioning

confidence: 99%

Simulation based perception testing for autonomous vehicles

Yang¹

View full text Add to dashboard Cite

Perception Testing technologies are widely applied in various scenarios, like industrial and academic research applications for autonomous driving systems. Accurate and robust autonomous driving simulation perception is pivotal for safety-guidance autonomous vehicles (AV). The autonomous vehicle system is facing the main challenges of a complex real-world environment with multi sensors' performance and their neighborhood view with an uncertain environment. The perception module exploits deep learning models to detect surrounding obstacles, including their types, positions, and velocities. However, the issue of LiDAR performance, which is the prediction of multi obstacle properties based on their deep learning model, remains unresolved. In addition, autonomous Vehicle systems rely on deep learning models to collect and modify raw point cloud data. It's crucial to check autonomous vehicles' robustness in several scenarios to generate automotive vehicles more safety and reliable. At this stage, it is easier to perform sensor integration to simulate sensors' performance for Autonomous Driving at various levels of verification. This thesis utilizes a realistic LiDAR point cloud and compares the difference between real-world and simulation environments to test perception modules for autonomous driving platform systems. This thesis proposes a simulation-based testing platform for autonomous vehicles to discover the potential shortage of perception by analysing huge scenarios and testing the suitable sensor performance in multi particular levels of the automated driving platforms. Additionally, this thesis introduces a simulation-based testing method, which involves multi-sensor configurations and includes virtual environment testing with various scenarios. To deal with real-time traffic scenarios, an effective way is utilized to search for the closest scenarios. Also, the proposed method has been tested in popular autonomous vehicle platforms and simulators . To utilize the existing methodology, we perform industry platforms on Baidu Apollo and guide it to assess the quality and enhance the perception system based on a simulation-based testing platform.

show abstract

Modeling Perception Errors towards Robust Decision Making in Autonomous Vehicles

Cited by 16 publications

References 14 publications

A Step Towards Efficient Evaluation of Complex Perception Tasks in Simulation

A Step Towards Efficient Evaluation of Complex Perception Tasks in Simulation

A Review of Testing Object-Based Environment Perception for Safe Automated Driving

Simulation based perception testing for autonomous vehicles

Contact Info

Product

Resources

About