Kubric: A scalable dataset generator

Abstract

Data is the driving force of machine learning, with the amount and quality of training data often being more important for the performance of a system than architecture and training details. But collecting, processing and annotating real data at scale is difficult, expensive, and frequently raises additional privacy, fairness and legal concerns. Synthetic data is a powerful tool with the potential to address these shortcomings: 1) it is cheap 2) supports rich ground-truth annotations 3) offers full control over data and 4) can circumvent or mitigate problems regarding bias, privacy and licensing. Unfortunately, software tools for effective data generation are less mature than those for architecture design and training, which leads to fragmented generation efforts. To address these problems we introduce Kubric, an open-source Python framework that interfaces with PyBullet and Blender to generate photo-realistic scenes, with rich annotations, and seamlessly scales to large jobs distributed over thousands of machines, and generating TBs of data. We demonstrate the effectiveness of Kubric by presenting a series of 13 different generated datasets for tasks ranging from studying 3D NeRF models to optical flow estimation. We release Kubric, the used assets, all of the generation code, as well as the rendered datasets for reuse and modification. segmentation depth optical flow surface normals object coordinatesFigure 1. Example scene created and rendered with Kubric along with some of the automatically generated annotations.

replicAnt: A pipeline for generating annotated images of animals in complex environments using Unreal Engine

“…Deep learning-based computer vision methods promise to fundamentally alter what is possible in animal behavioural research [12][13][14][15][16][17][18]. A key remaining bottleneck is the "data-hunger" of supervised learning techniques: annotated datasets of the size and variability required to achieve robust, domain-invariant performance are rarely available, and in any case time-intensive to produce [36,54]. One strategy to overcome this limitation is to produce annotated data synthetically, using sufficiently realistic computer simulations [31,32,34,36].…”

“…A key remaining bottleneck is the "data-hunger" of supervised learning techniques: annotated datasets of the size and variability required to achieve robust, domain-invariant performance are rarely available, and in any case time-intensive to produce [36,54]. One strategy to overcome this limitation is to produce annotated data synthetically, using sufficiently realistic computer simulations [31,32,34,36]. In order to facilitate this process, we developed replicAnt: a synthetic data generator built in Unreal Engine 5 and Python.…”

“…In robotic [34][35][36], human [37][38][39][40], and automated driving [41][42][43] applications, annotated datasets comprising billions of images can now be produced "synthetically", i. e. through simulation with a computer. By placing 3D models in simulated environments, variable and annotated datasets can be generated at scale, and at a fraction of the cost and time required for hand-annotation of real images [39,40,42,44]. The use of synthetic data is particularly attractive where annotated real datasets are practically absent or only of insufficient size, as is the case for almost all non-human animal studies [22,[30][31][32][45][46][47][48][49][50][51].…”

Object-Centric Slot Diffusion

“…We evaluate our model on five datasets. Four of them are synthetic multi-object datasets-CLEVR [35], CLEVRTex [37], MOVi-C, MOVi-E [24]. They present increasing levels of difficulty-CLEVRTex adds texture to objects and backgrounds, MOVi-C uses more complex objects and natural backgrounds, and MOVi-E contains large numbers of objects (up to 23) per scene.…”

BlenderProc2: A Procedural Pipeline for Photorealistic Rendering

“…A particular focus is placed on the acknowledgment of the simulation-to-real gap and how to tackle this particular challenge in the dataset generation process. Even though the first version of BlenderProc was one of the first tools to generate photo-realistic, synthetic datasets, many more tools exist nowadays, compared in Table 1 (Greff et al, 2022;Manolis Savva* et al, 2019;Morrical et al, 2021;Schwarz & Behnke, 2020;To et al, 2018). In contrast to the first version of BlenderProc, BlenderProc2 relies on an easy-to-use python API, whereas the first version used a YAML-based configuration approach (Denninger et al, 2019(Denninger et al, , 2020.…”