Generic Simulation Framework for Evaluation Process: Applied to AI-powered Visual Perception System in Autonomous Driving

Abstract

The rapid development of modern vehicle technology necessitates a robust and comprehensive evaluation framework for AI-powered systems in Autonomous Driving Systems (ADS). This paper introduces a generic simulation framework tailored to the evaluation process of AI-powered systems in ADS, with an emphasis on the visual perception system as a case study. The framework integrates different levels of evaluation that contain critical metrics, providing a comprehensive understanding of system performance and targeted evaluation and analysis of components, under challenging driving conditions. This information can be used to guide improvements to the system, such as selecting better AI algorithms, modifying the design of the system, or improving the environment in which it operates. For each component, the framework not only incorporates a unified definition and configuration but also establishes a communication mechanism, which contributes to effectively integrating different tools and platforms into the evaluation process.The insights gleaned from the visual perception systems case study can be leveraged for adapting and applying the generic evaluation process to other AI-powered systems in ADS, aiming to promote the development of more reliable and safer systems.