Benchmarking and Functional Decomposition of Automotive Lidar Sensor Models

Abstract

Simulation-based testing is seen as a major requirement for the safety validation of highly automated driving. One crucial part of such test architectures are models of environment perception sensors such as camera, lidar and radar sensors. Currently, an objective evaluation and the comparison of different modeling approaches for automotive lidar sensors are still a challenge. In this work, a real lidar sensor system used for object recognition is first functionally decomposed. The resulting sequence of processing blocks and interfaces is then mapped onto simulation methods. Subsequently, metrics applied to the aforementioned interfaces are derived, enabling a quantitative comparison between simulated and real sensor data at different steps of the processing pipeline. Benchmarks for several existing sensor models at a concrete selected interface are performed using those metrics by comparing them to measurements gained from the real sensor. Finally, we outline how metrics on low-level interfaces can correlate with results on more abstract ones. A major achievement of this work lies within the commonly accepted interfaces and a common understanding of real and virtual lidar sensor systems and, even more important, an initial guideline for the quantitative comparison of sensor models with the ambition to support future validation of virtual sensor models.

Sequential lidar sensor system simulation: a modular approach for simulation-based safety validation of automated driving

“…There are multiple tools available (commercial or opensource) that already provide parameterizable models to generate either synthetic PCLs or OLs with different levels of fidelity [6]. Actual methods for lidar sensor simulation to generate a synthetic PCL are either ray casting / tracing methods or projection methods like "z-buffer" as described and compared in the previous work of the authors [1]. For simulation of lidar-typical OLs, it is common sense that a stochastic or phenomenological approach is sufficient to generate desired uncertainties of existence (e.g., FP-/ FNrates), states (e.g., bounding box dimensions / location), and classes, as they are described by Dietmayer [7].…”

“…Besides the already described limitation in fidelity, actual stochastic single-solution OL simulation lacks further Fig. 1 A generic perception sensor system for object detection along with the interface (IF) definition adapted from [1]. After signal processing, the data from one or more sensors can be merged on IF1 by the alignment and fusion module.…”

“…3 Modules of the sequential lidar sensor system simulation Figure 5 shows the sequential structure of the lidar system simulation approach using functional blocks and interfaces as presented in earlier work of the authors [1]. When generating an OL, the environment simulation provides global GT data, often in a world-coordinate frame.…”

“…In this regard, it is differentiated between sensor frontend and sensor system to distinguish signal transmitting, perception, and pre-processing (the front-end) from data processing [1], as can be seen in Fig. 1.…”

“…The modules themselves are found through functional decomposition, as shown in the previous work of the authors [1]: Typical automotive lidar sensors output detections as a point cloud (PCL), which is an ID-ordered distance and intensity tuple at IF1 that is also the output of the sensor model. IF2 is part of the data processing unit before, e.g., object tracking and classification takes place, as it provides features in Cartesian coordinates in the vehicle coordinate system [ [2], p. 2], e.g., the output of a sensor fusion unit that combines two lidar sensors into one PCL.…”