Testing ground-truth errors in an automotive dataset for a DNN-based object detector

Abstract

Given the promising advances in the field of Assisted and Automated Driving, it is expected that the roads of the future will be populated by vehicles driven by computers, partially or fully replacing human drivers. In this scenario, the first stage of the perception-decision-actuation pipeline will likely rely on Deep Neural Networks for understanding the scene around the vehicle. Typical tasks for Deep Neural Networks are object detection and instance segmentation, tasks relying on supervised learning and annotated datasets. As one can imagine, the quality of the labelled dataset strongly affects the performance of the network, and this aspect is investigated in this paper. Annotation quality should be a primary concern in safety-critical tasks, such as Assisted and Automated Driving. This work addresses and classifies some of the mistakes found in a popular automotive dataset. Moreover, some experiments with a Deep Neural Network model were performed to test the effect of these mistakes on network predictions. A set of criteria was established to support the relabelling of the testing dataset which was compared to the original dataset.

Influence of AVC and HEVC Compression on Detection of Vehicles Through Faster R-CNN

“…When reviewing the KITTI dataset, this kind of mismatched ground truth was present in a significant number of evaluated images. There were also mislabels in which vehicles fully obstructed by another object were still labelled in the ground truth, as further discussed in [46].…”

Run-Time Introspection of 2D Object Detection in Automated Driving Systems Using Learning Representations

“…Furthermore, as seen in the first row of Figure 7 there are some objects far away neither highlighted as true positive or false negative. This is due to the labelling of the datasets, which sometimes might be erroneous or partial [39]. It is also possible that such cases may confuse the introspection mechanisms as the activations may indicate the presence of an object in those areas.…”

Impact of the ground truth quality for handwriting recognition

“…In related fields, several studies have been conducted to investigate the impact of ground truth quality on deep learning, for example in the context of object detection [2,13], text-line segmentation [3,22], and semantic segmentation [20,25] in natural images or historical document images. However, the problems encountered for HTR are specific and to the best of our knowledge, there are currently no comprehensive studies on the impact of ground-truth quality for deep learning-based HTR.…”