Quality Metrics and Oracles for Autonomous Vehicles Testing

Abstract

The race for deploying AI-enabled autonomous vehicles (AVs) on public roads is based on the promise that such self-driving cars will be as safe as or safer than human drivers. Numerous techniques have been proposed to test AVs, which however lack oracle definitions that account for the quality of driving, due to the lack of a commonly used set of metrics. Towards filling this gap, we first performed a systematic analysis of the literature concerning the assessment of the quality of driving of human drivers and extracted 126 metrics. Then, we measured the correlation between such metrics and the human perception of driving quality when AVs are driving. Lastly, we performed a study based on mutation analysis to assess whether the 26 metrics that best capture the quality of AV driving according to the human study can be used as functional oracles. Our results, targeting the Udacity platform, indicate that our automated oracles can kill a high proportion of mutants at a zero or very low false alarm rate, and therefore can be used as effective functional oracles for the quality of driving of AVs.

Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems

“…We assess and compare the quality of driving by analyzing the distributions of three metrics, namely steering angle, lateral deviation, and predictive uncertainty. The first two metrics are proposed by Jahangirova et al [10] as effective metrics to evaluate the lane-keeping capability of SDC models. The last metric is used in the self-driving car domain to account for the DNN model's confidence [37].…”

“…In our setup, the car follows the middle line on a two-lane road (as if it were a single-lane, one-way road) and moves only forward. Thus, the lateral position is regarded as the key telemetry value to assess the lane-keeping capability of SDC models, as shown by a recent study [10].…”

“…In our virtual platform, we manually placed an ordered sequence of 22 waypoints onto the simulated track along the center of the road, to allow tracking the XTE of the vehicle during motion. Then, the simulator measures the lateral position [10], or cross-track error (XTE), as the distance between the car's cruising position and the segment between the two closest consecutive waypoints [38].…”

“…Specific, system-level, testing solutions for SDCs have been proposed [6], [7], [10], [11], [12], [13], [14], [15], in which the dominant approach for the empirical evaluation relies on the use of simulators, which does not guarantee that the observed failures, or lack thereof, correlate with those observable during on-road testing. Recent works have confirmed the need for real-world testing for robotic systems, as simulation platforms are often decoupled from the real world complexities [16], [17].…”

“…We compared the testing results obtained in the virtual vs the real world, in terms of quality of driving and exposed failures. First, we assessed the transferability of driving behavior, using common driving quality metrics [10]. We found that steering angle predictions are statistically indistinguishable, whereas the SDCs' trajectories and the DNN models' uncertainties have non-negligible differences across the virtual and realworld environment.…”