Systematization of Relevant Road Users for the Evaluation of Autonomous Vehicle Perception

Philipp, Robin; Rehbein, Jana; Grün, Felix; Hartjen, Lukas; Zhu, Zhijing; Schuldt, Fabian; Howar, Falk

doi:10.1109/syscon53536.2022.9773877

Cited by 6 publications

(2 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While not typically included in benchmarks, other metrics considering the driving task and its safety have been proposed. While they are not within the scope of this work, several works such as [6], [71], [76], [91], [95] attempt to focus the perception evaluation on objects relevant for collision avoidance.…”

Section: Safety Aware Metricsmentioning

confidence: 99%

SHARD: Safety and Human Performance Analysis for Requirements in Detection

Mori,

Peters

2024

IEEE Trans. Intell. Veh.

View full text Add to dashboard Cite

Automated driving requires reliable perception of the environment to ensure the safety of the driving task. One common perception task is 3D object detection, which aims at perceiving location and attributes of dynamic objects. This task is typically evaluated on different benchmark datasets, which each propose different metrics. However, these different metrics generally lack consistency and bear no relation to safety. Most notably, there is a lack of consistent definitions of pass/fail criteria for any given detection metric. In this work, the issue is addressed by systematically considering safety and human performance across different aspects of the object detection task. This approach yields interpretable detection metrics as well as thresholds for pass/fail criteria. Furthermore, a validation approach leveraging a prediction network is introduced and successfully applied to the requirements. A comparison of existing detectors shows that current perception algorithms exhibit failures for a majority of objects on the nuScenes dataset. Therefore, the results indicate the necessity of explicit safety consideration in the development of perception algorithms for the automated driving task.

show abstract

Section: Safety Aware Metricsmentioning

confidence: 99%

SHARD: Safety and Human Performance Analysis for Requirements in Detection

Mori,

Peters

2024

IEEE Trans. Intell. Veh.

View full text Add to dashboard Cite

show abstract

“…At this point, there is no standardized evaluation metric for autonomous vehicles. Some new evaluation metrics that cover a few of the problems have recently been suggested [54], [55], [56]. However, none of them could establish as a new standard, since not a single one covers all of the problems.…”

Section: A Perception System Evaluationmentioning

confidence: 99%

Learn to See Fast: Lessons Learned From Autonomous Racing on How to Develop Perception Systems

et al. 2023

View full text Add to dashboard Cite

The objective of this work is to provide a comprehensive understanding of the development of autonomous vehicle perception systems. So far, most autonomy perception research has been concentrated on improving perception systems' algorithmic quality or combining different sensor setups. In our work, we draw conclusions from participating in the Indy Autonomous Challenge 2021 and its follow-up event in Las Vegas 2022. These were the first head-to-head autonomous racing competitions that required an entire perception pipeline to perceive the environment and the opposing surrounding vehicles. Our research includes quantitative results from collected vehicle data and qualitative results from simulation, video, and multiple race analysis. The Indy Autonomous Challenge was one of the few research projects that considered the entire autonomous vehicle. Therefore, our findings indicate insights on the system level, including hardware setup and full-stack software. We can demonstrate that different sensor modalities in the vehicle have strengths and weaknesses when they are deployed. Our results further show the difficulties and challenges that emerge when multi-modal perception systems must run in real-time on real-world autonomous vehicles. The most concise finding from our investigation is the summary of critical learnings when developing and deploying perception systems for autonomous systems. Given the background of the study, it was inevitable that our conclusions were influenced by driving on the racetrack and only one hardware setup available. Therefore, in the discussion, we draw further parallels to driving on public roads in dense traffic. More studies are needed to investigate the development and deployment of multi-modal perception systems for autonomous road vehicles with different hardware setups and various object detection, localization, and prediction algorithms. The novel contributions of this work are given by 12 lessons learned, summarized in 5 categories. These were derived and validated through a realized real-world application project. The videos of the final events in Indianapolis and Las Vegas can be watched here: IAC: https://www.youtube.com/watch?v=ERTffn3IpIs&ab_channel=CNETHighlights AC@CES: https://www.youtube.com/watch?v=df9f4Qfa0uU&ab_channel=CNETHighlights Multiple modules of the software stack are open source: https://github.com/TUMFTM.INDEX TERMS Autonomous racing, autonomous vehicles, perception systems, software development.The associate editor coordinating the review of this manuscript and approving it for publication was Junho Hong . FIGURE 2. Main contribution of this work: the lessons learned presented.to develop, deploy, validate, and test a complete autonomous driving stack, from sensor data to vehicle actuation. Because of the challenge's tight schedule, software development had to begin before the vehicles were built. This made the simulation of the vehicles and the environment an important task to be successful in the final competition.The IAC took place on October 23, 2021, i...

show abstract