Image-Based Synthesis and Re-synthesis of Viewpoints Guided by 3D Models

Abstract

We propose a technique to use the structural information extracted from a set of 3D models of an object class to improve novel-view synthesis for images showing unknown instances of this class. These novel views can be used to "amplify" training image collections that typically contain only a low number of views or lack certain classes of views entirely (e. g. top views).We extract the correlation of position, normal, reflectance and appearance from computer-generated images of a few exemplars and use this information to infer new appearance for new instances. We show that our approach can improve performance of state-of-the-art detectors using real-world training data. Additional applications include guided versions of inpainting, 2D-to-3D conversion, superresolution and non-local smoothing.

On rendering synthetic images for training an object detector

“…PASCAL VOC 2007. We use this well-known Computer Vision benchmark to compare our approach to a very recent work on synthetic view generation [11]. As in [11], we restrict ourselves here to the car class, which nevertheless further demonstrates the versatility of our approach.…”

“…A more sophisticated model could be used to address this issue. However realism does not seem to be critical for our purposes since our simplistic model is sufficient to outperform [11]. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.…”

“…We compare here our approach with the one of [11], which was applied to car detection on the PASCAL VOC car dataset. Like ours, it uses a CAD model of the target object and seed real images.…”

“…Like ours, the approach of [11] relies on both real training images and a 3D model. After registering the 3D model to the images, the material and lighting properties of the different object components are estimated and used to synthesise new views of the 3D model.…”

Analysis of 3D Face Modeling

“…3D range of scans in a cylindrical parameterization r(h, φ) [26]. Konstantinos Rematas's [27] work on the synthesis of images and improved, novel view of unknown instances of this class. Additional applications were 2D-to-3D conversion, super resolution and non-local smoothing.…”