Challenges in Designing Datasets and Validation for Autonomous Driving

Abstract

Autonomous driving is getting a lot of attention in the last decade and will be the hot topic at least until the first successful certification of a car with Level 5 autonomy (International, 2017). There are many public datasets in the academic community. However, they are far away from what a robust industrial production system needs. There is a large gap between academic and industrial setting and a substantial way from a research prototype, built on public datasets, to a deployable solution which is a challenging task. In this paper, we focus on bad practices that often happen in the autonomous driving from an industrial deployment perspective. Data design deserves at least the same amount of attention as the model design. There is very little attention paid to these issues in the scientific community, and we hope this paper encourages better formalization of dataset design. More specifically, we focus on the datasets design and validation scheme for autonomous driving, where we would like to highlight the common problems, wrong assumptions, and steps towards avoiding them, as well as some open problems.

WoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving

“…when the sought model does not require hyper-parameter selection) or be used for model selection, and finally, the testing set is used for model evaluation purposes only. The dataset supports correct hypothesis evaluation [52], therefore multiple splits are provided (5 in total). Depending on the particular task (see Section 4, for the full list), the class imbalance may be an issue [17], therefore, task-specific splits are also provided.…”

Near-field Perception for Low-Speed Vehicle Automation using Surround-view Fisheye Cameras

“…Automotive scenes are very diverse and the system is expected to work across countries as well as varying weather and lighting conditions. One of the main challenges is to build an effective dataset which covers diverse aspects [55]. CNNs are computationally intensive and efficient design techniques are critical to be incorporated [56], [57].…”

TiledSoilingNet: Tile-level Soiling Detection on Automotive Surround-view Cameras Using Coverage Metric

“…Our dataset creation strategy remains the same as discussed in [18], [21]. For this experiment, a total of 1, 05, 987 images from all four cameras around the car was collected.…”