A Survey of Image Synthesis Methods for Visual Machine Learning

Abstract

Image synthesis designed for machine learning applications provides the means to efficiently generate large quantities of training data while controlling the generation process to provide the best distribution and content variety. With the demands of deep learning applications, synthetic data have the potential of becoming a vital component in the training pipeline. Over the last decade, a wide variety of training data generation methods has been demonstrated. The potential of future development calls to bring these together for comparison and categorization. This survey provides a comprehensive list of the existing image synthesis methods for visual machine learning. These are categorized in the context of image generation, using a taxonomy based on modelling and rendering, while a classification is also made concerning the computer vision applications they are used. We focus on the computer graphics aspects of the methods, to promote future image generation for machine learning. Finally, each method is assessed in terms of quality and reported performance, providing a hint on its expected learning potential. The report serves as a comprehensive reference, targeting both groups of the applications and data development sides. A list of all methods and papers reviewed herein can be found at https://computergraphics.on.liu.se/image_synthesis_methods_for_visual_machine_learning/.

Learning from Scarce Information: Using Synthetic Data to Classify Roman Fine Ware Pottery

“…The rationale behind using different packages to produce simulated images was to investigate how sensitive the overall approach is to different degrees of realism reflected in the simulated data, with Matplotlib generating coarse images of pots and Blender enabling the generation of realistically looking images (see [ 51 ] for the entire dataset we produced using Matplotlib and Blender ). For a comprehensive overview of other image synthesis methods we refer the reader to [ 54 ].…”

Generating Images with Physics-Based Rendering for an Industrial Object Detection Task: Realism versus Domain Randomization

“…Compared to DR, which randomizes the full content of the simulated environment, PBR uses rendering randomization. PBR tries to simulate reality as close as possible while also randomizing environment parameters, such as lighting and the virtual camera’s position, to generate diverse training data [ 25 ]. Hodaň et al [ 12 ] used the path tracing render engine Arnold [ 26 ] to generate highly photorealistic images in order to train a Faster-RCNN object detector.…”

VIS30K: A Collection of Figures and Tables from IEEE Visualization Conference Publications

“…These pseudo-papers are sufficient to guide CNNs to detect and localize the figures and tables from real documents. While the simulation approach has been used in other realistic environments [42], ours is to our knowledge the first use for scholarly document analysis. Our approach leverages the simple assumption that the form and structural content of a page are more important for detecting images than the factual content.…”