Metamorphic Testing of Advanced Driver-Assistance Systems: Implementing Euro NCAP Standards on OpenStreetMap

Abstract

Simulation testing is considered the supplement to ensure the safety of Autonomous driving (AD) and advanced driving assistance (ADAS's) systems in terms of time and costs. However, it is very difficult and challenging when the simulation results are unexpected. This work presents a simulation-based metamorphic testing (MT) approach to test the ADAS system, implementing the European new car assessment program (Euro NCAP) standards on OpenStreetMap (OSM). We first defined input patterns and relations related to autonomous driving, following the principles of MT. To assess the approach, we executed three tests in two steps at both the design and system levels. Our results show that none of the three (source) tests detected any collisions. However, for follow-up test cases, the ego vehicle failed to apply brakes to avoid a collision when the speed changed. A real-life issue in the system was immediately revealed and confirmed by the development team. We then designed a mechanism and continued the test to check whether the recorded collisions were avoidable. Our results (rate of 5.8%) indicate the fault detection effectiveness when testing the ADAS system. Although we applied the approach to testing the ADAS driving performance, it can be applied to other AD systems. This research, therefore, provides a systematic way to design and test autonomous driving technologies and integrate testing standards with metamorphic testing.