On rendering synthetic images for training an object detector

Rozantsev, Artem; Lepetit, Vincent; Fua, Pascal

doi:10.1016/j.cviu.2014.12.006

Cited by 105 publications

(50 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While most works use either real or synthetic data, only few papers consider the problem of training deep models with mixed reality. [22] estimate the parameters of a rendering pipeline from a small set of real images for training an object detector. [10] use synthetic data for text detection in images.…”

Section: Related Workmentioning

confidence: 99%

Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes

et al. 2018

View full text Add to dashboard Cite

The success of deep learning in computer vision is based on the availability of large annotated datasets. To lower the need for hand labeled images, virtually rendered 3D worlds have recently gained popularity. Unfortunately, creating realistic 3D content is challenging on its own and requires significant human effort. In this work, we propose an alternative paradigm which combines real and synthetic data for learning semantic instance segmentation and object detection models. Exploiting the fact that not all aspects of the scene are equally important for this task, we propose to augment real-world imagery with virtual objects of the target category. Capturing real-world images at large scale is easy and cheap, and directly provides real background appearances without the need for creating complex 3D models of the environment. We present an efficient procedure to augment these images with virtual objects. This allows us to create realistic composite images which exhibit both realistic background appearance as well as a large number of complex object arrangements. In contrast to modeling complete 3D environments, our data augmentation approach requires only a few user interactions in combination with 3D shapes of the target object category. Through an extensive set of experiments, we conclude the right set of parameters to produce augmented data which can maximally enhance the performance of instance segmentation models. Further, we demonstrate the utility of proposed approach on training standard deep models for semantic instance segmentation and object detection of cars in outdoor driving scenarios. We test the models trained on our augmented data on the KITTI 2015 dataset, which we have annotated with pixelaccurate ground truth, and on the Cityscapes dataset. Our experiments demonstrate that models trained on augmented imagery generalize better than those trained on synthetic data or models trained on limited amounts of annotated real data.

show abstract

Section: Related Workmentioning

confidence: 99%

Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes

et al. 2018

View full text Add to dashboard Cite

show abstract

“…In general, data creation is a time consuming and expensive process which requires huge human efforts. More recently, an alternative form of data generation process with minimal supervision is getting popular [32,61,66], which uses synthetic mechanisms to render and annotate images in an appropriate form. The simple idea of generating data synthetically allows overcoming the challenges in obtaining the data.…”

Section: Handwritten Synthetic Datasetmentioning

confidence: 99%

HWNet v2: an efficient word image representation for handwritten documents

Krishnan

Jawahar

2019

IJDAR

View full text Add to dashboard Cite

We present a framework for learning an efficient holistic representation for handwritten word images. The proposed method uses a deep convolutional neural network with traditional classification loss. The major strengths of our work lie in: (i) the efficient usage of synthetic data to pre-train a deep network, (ii) an adapted version of the ResNet-34 architecture with the region of interest pooling (referred to as HWNet v2) which learns discriminative features for variable sized word images, and (iii) a realistic augmentation of training data with multiple scales and distortions which mimics the natural process of handwriting. We further investigate the process of transfer learning to reduce the domain gap between synthetic and real domain, and also analyze the invariances learned at different layers of the network using visualization techniques proposed in the literature.Our representation leads to a state-of-the-art word spotting performance on standard handwritten datasets and historical manuscripts in different languages with minimal representation size. On the challenging iam dataset, our method is first to report an mAP of around 0.90 for word spotting with a representation size of just 32 dimensions. Furthermore, we also present results on printed document datasets in English and Indic scripts which validates the generic nature of the proposed framework for learning word image representation.

show abstract

“…To our knowledge, learning from synthetic RGB-D images for pixel-wise object segmentation tasks has yet received little attention. In the context of generic object detection and recognition, a first kind of approach consists in generating synthetic images by projecting 3D models onto natural "background" images [20,28]. However, these synthetic images contain by construction at most a few object instances that are implicitly assumed to be distinguishable from their environment, thus excluding many situations like the present case of dense stack of object instances.…”

Section: Synthetic Training Datamentioning

confidence: 99%

Object Segmentation in Depth Maps with One User Click and a Synthetically Trained Fully Convolutional Network

Grard

Brégier

Sella³

et al. 2018

Springer Proceedings in Advanced Robotics

View full text Add to dashboard Cite

With more and more household objects built on planned obsolescence and consumed by a fast-growing population, hazardous waste recycling has become a critical challenge. Given the large variability of household waste, current recycling platforms mostly rely on human operators to analyze the scene, typically composed of many object instances piled up in bulk. Helping them by robotizing the unitary extraction is a key challenge to speed up this tedious process. Whereas supervised deep learning has proven very efficient for such object-level scene understanding, e.g., generic object detection and segmentation in everyday scenes, it however requires large sets of per-pixel labeled images, that are hardly available for numerous application contexts, including industrial robotics. We thus propose a step towards a practical interactive application for generating an object-oriented robotic grasp, requiring as inputs only one depth map of the scene and one user click on the next object to extract. More precisely, we address in this paper the middle issue of object segmentation in top views of piles of bulk objects given a pixel location, namely seed, provided interactively by a human operator. We propose a two-fold framework for generating edge-driven instance segments. First, we repurpose a state-of-the-art fully convolutional object contour detector for seed-based instance segmentation by introducing the notion of edge-mask duality with a novel patch-free and contour-oriented loss function. Second, we train one model using only synthetic scenes, instead of manually labeled training data. Our experimental results show that considering edge-mask duality for training an encoder-decoder network, as we suggest, outperforms a state-of-the-art patch-based network in the present application context.

show abstract

On rendering synthetic images for training an object detector

Cited by 105 publications

References 30 publications

Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes

Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes

HWNet v2: an efficient word image representation for handwritten documents

Object Segmentation in Depth Maps with One User Click and a Synthetically Trained Fully Convolutional Network

Contact Info

Product

Resources

About