Playing for Data: Ground Truth from Computer Games

Abstract

Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixellevel labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 1 3 of the CamVid training set outperform models trained on the complete CamVid training set.

Comparison of Visual Datasets for Machine Learning

“…This laborious process significantly slows down the creation of a dataset and could also affect the selection of the data. Some researchers suggest using computer graphics to create labels [10], but graphics technologies do not always generate "photo-realistic" images and videos.…”

A Survey on Visual Traffic Simulation: Models, Evaluations, and Applications in Autonomous Driving

“…[CSKX15] use TORCS [WEG*00] to evaluate the proposed direct perception model for autonomous driving. Recently, researchers [RVRK16, JRBM*17, RHK17] leverage Grand Theft Auto V (GTA V) to derive autonomous driving policies, which result in comparable performance to control policies that derived from manually annotated real‐world images.…”

Evaluation of a Traffic Sign Detector by Synthetic Image Data for Advanced Driver Assistance Systems

“…from a driving simulator). For example, they state that "realism" is achieved by "the high fidelity of material appearance and light transport simulation" and "the content of the virtual worlds", which includes "the layout of objects and environments, the realistic textures, the motion of vehicles and autonomous characters, the presence of small objects that add detail, and the interaction between the player and the environment (Richter et al, 2016). More details, for example which criteria define a high-fidelity material, are not given and subsequently it is also not proven whether GTA V fulfills these criteria.…”

“…The datasets are, ordered by the date of publication of the corresponding paper: the Virtual KITTI (further abbreviated as vKITTI) dataset from (Gaidon et al, 2016), the SYNTHetic collection of Imagery and Annotations (SYNTHIA) dataset from (Ros et al, 2016), the unnamed dataset from (Richter et al, 2016), inspired by the title of the paper, Playing for Data, further called PfD, the unnamed dataset from (Johnson-Roberson et al, 2016), further called DitM, also inspired by the title of the paper, Driving in the Matrix, and the VIsual PERception (VIPER) dataset from (Richter et al, 2017).…”

“…creating and quality-checking one high-quality semantic ground truth image for Cityscapes required more than 90 minutes (Cordts et al, 2016)) and prone to errors made by the lablers (e.g. the Cityscapes dataset has in average an annotation density of 97.1% of all pixels of the high-quality semantic ground truth images (Richter et al, 2016)). …”