The Synthinel-1 dataset: a collection of high resolution synthetic overhead imagery for building segmentation

Abstract

Recently deep learningnamely convolutional neural networks (CNNs)have yielded impressive performance for the task of building segmentation on large overhead (e.g., satellite) imagery benchmarks. However, these benchmark datasets only capture a small fraction of the variability present in real-world overhead imagery, limiting the ability to properly train, or evaluate, models for real-world application. Unfortunately, developing a dataset that captures even a small fraction of real-world variability is typically infeasible due to the cost of imagery, and manual pixel-wise labeling of the imagery. In this work we develop an approach to rapidly and cheaply generate large and diverse virtual environments from which we can capture synthetic overhead imagery for training segmentation CNNs. Using this approach, generate and publicly-release a collection of synthetic overhead imagery termed Synthinel-1 with full pixel-wise building labels. We use several benchmark dataset to demonstrate that Synthinel-1 is consistently beneficial when used to augment real-world training imagery, especially when CNNs are tested on novel geographic locations or conditions.

Leveraging Synthetic Data in Object Detection on Unmanned Aerial Vehicles

“…However, their world is lifeless and does not feature any objects of interest. The Synthinel-1 [34] data set features synthetic data showing building footprints with segmentation masks.…”

UnityShip: A Large-Scale Synthetic Dataset for Ship Recognition in Aerial Images

“…Within the last few years, there has been some research that has focused on applications of synthetic data in aerial imagery. For example: Bondi et al [46] used the AirSim tool to generate BIRDSAI, an infrared object recognition dataset that blends both real and synthetic data; Chen et al [47] published VALID, a synthetic dataset for instance segmentation, semantic segmentation, and panoramic segmentation from an unmanned aircraft perspective; Kong et al [48] investigated the application of synthetic data to semantic segmentation for remote sensing and published the Synthinel-1 dataset; Shermeyer et al [49] released RarePlanes, a hybrid real-and synthetic-image dataset for aircraft detection and fine-grained attribute recognition in aerial imagery; Clement et al [50] also released a synthetic image dataset for aircraft recognition, and this is mainly intended for rotated object recognition; Lopez-Campos et al [51] released ESPADA, the first synthetic dataset for depth estimation in aerial images; Uddin et al [52] proposed a method for converting optical videos to infrared videos using GAN, investigating the impact for classification and detection task.…”

Randomized Histogram Matching: A Simple Augmentation for Unsupervised Domain Adaptation in Overhead Imagery

“…One emergent limitation of DNNs in remote sensing however is their sensitivity to the statistics of their training imagery. Recent research has shown that DNNs often perform unpredictably, and often much more poorly, when they are applied to novel collections of imagery, which were not present in their training data [31,19,22,14]. Furthermore, this performance degradation seems to occur even if DNNs are trained on relatively large and diverse datasets, encompassing large and diverse geographic regions [19,14].…”

“…The performance degradation of DNNs on new sets of imagery are caused by visual domain shift (a.k.a. distribution shift): statistical differences between the training imagery and new collections of imagery [31,19]. Fig.…”

“…One straightforward solution to these domain shifts is to label a subset of each new collection of imagery and then re-train the DNN; however, this solution is costly and timeconsuming [31,19]. We would instead ideally have a model that performs well across many different collections of imagery, and does so without the need for labels from each one.…”