Automotive Perception System Evaluation with Reference Data from a UAV’s Camera Using ArUco Markers and DCNN

Abstract

Testing and evaluation of an automotive perception system is a complicated task which requires special equipment and infrastructure. To compute key performance indicators and compare the results with real-world situation, some additional sensors and manual data labelling are often required. In this article, we propose a different approach, which is based on a UAV equipped with a 4K camera flying above a test track. Two computer vision methods are used to precisely determine the positions of the objects around the car – one based on ArUco markers and the other on a DCNN (we provide the algorithms used on GitHub). The detections are then correlated with the perception system readings. For the static and dynamic experiments, the differences between various systems are mostly below 0.5 m. The results of the experiments performed indicate that this approach could be an interesting alternative to existing evaluation solutions.

Research Scenarios of Autonomous Vehicles, the Sensors and Measurement Systems Used in Experiments

“…− emergency braking before an obstacle [34,71,72], where the aim is to minimize the risk of collision; − avoiding a suddenly appearing obstacle in a critical situation where it is not possible to stop the AV [34,72]; − 2D object detection, localization, and tracking: car detection [70,73,74] and distance estimation to other vehicles [27,75]; pedestrian detection [20,21,69]; bicyclist/motorcyclist detection [20,73,74].…”

“…− 3D object detection on the road [72,76] using lidar and sensor fusion (e.g., lidar and camera), where the aim is to obtain information about the obstacle in the spatial coordinate system: car detection [19,75,77]; pedestrian detection [23,24,78]; bicyclist/motorcyclist detection [23,25,73]; cone detection [67]; detection of walls, trees, bushes, and poles [78]; detection and tracking of unknown objects [79].…”

“…Their widest area of application is object detection on roads. There are research scenarios that take into account 2D detection of objects and obstacles [20,70,75]. There are separate scenarios for the detection and localization of objects in 3D space, based on the sensory fusion of several perception sensors (cameras with lidars or radars) [24,67,77].…”

“…In [19,33], the GPS sensor is used in path planning and object detection scenarios. A similar application has the GNSS RTK Swift Navigation sensor in [31,75]. This sensor is used in lane keeping, localization, and object tracking scenarios.…”

“…[33,34] Lane keeping by the AV, 2D and 3D object detection, localization, tracking GNSS RTK SwiftNavigation. [31,75] 2D and 3D object detection, localization, tracking IMU Vectornav VN-100. Single RTK GPS Trimble AG GPS361.…”