Semi-Parametric Image Synthesis

Qi, Xiaojuan; Chen, Qifeng; Jia, Jiaya; Koltun, Vladlen

doi:10.1109/cvpr.2018.00918

Cited by 184 publications

(185 citation statements)

References 32 publications

Supporting

Mentioning

185

Contrasting

Order By: Relevance

“…Non-Parametric Visual Manipulation Non-parametric image synthesis approaches [18,6,27] usually generate new images by warping and stitching together existing patches from a database. The idea is extended by Qi et al [37] which combines neural networks to improve quality. Though similar at first glance, our method is intrinsically different from non-parametric image synthesis: our local embedding sub-network encodes facial instances as embeddings instead of image patches.…”

Section: Related Workmentioning

confidence: 99%

Mask-Guided Portrait Editing With Conditional GANs

Bao

Yang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

143

View full text Add to dashboard Cite

a) Mask2image (b) Component editing (c) Component transfer Figure 1: We propose a framework based on conditional GANs for mask-guided portrait editing. (a) Our framework can generate diverse and realistic faces using one input target mask (lower left corner in the first image). (b) Our framework allows us to edit the mask to change the shape of face components, i.e. mouth, eyes, hair. (c) Our framework also allows us to transfer the appearance of each component for a portrait, including hair color. AbstractPortrait editing is a popular subject in photo manipulation. The Generative Adversarial Network (GAN) advances the generating of realistic faces and allows more face editing. In this paper, we argue about three issues in existing techniques: diversity, quality, and controllability for portrait synthesis and editing. To address these issues, we propose a novel end-to-end learning framework that leverages conditional GANs guided by provided face masks for generating faces. The framework learns feature embeddings for every face component (e.g., mouth, hair, eye), separately, contributing to better correspondences for image translation, and local face editing. With the mask, our network is available to many applications, like face synthesis driven by mask, face Swap+ (including hair in swapping), and local manipulation. It can also boost the performance of face parsing a bit as an option of data augmentation.

show abstract

Section: Related Workmentioning

confidence: 99%

Mask-Guided Portrait Editing With Conditional GANs

Bao

Yang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

143

View full text Add to dashboard Cite

show abstract

“…It may also be due to design of architecture and hyper-parameters specifically suited for Cityscapes, and that efforts are required to tune hyper-parameters to make it work for a large and diverse dataset as COCO. It is for this reason we also use Cityscapes to contrast our approach with prior works [9,32,43,54] for the sake of fair comparison. Additionally, we resize our generated outputs to 256 × 256 just to make a fair comparison with Pix2Pix on COCO.…”

Section: Methodsmentioning

confidence: 99%

“…while our approach is most likable, there are still many situations where our approach produced undesirable outputs. Cityscapes: Table 3 contrasts the performance of our approach with prior approaches [9,32,43,54] that have specifically demonstrated on Cityscapes. Except Pix2Pix, we used publicly available results for this evaluation.…”

Section: Mask-rcnn Scoresmentioning

confidence: 99%

“…Non-parametric methods [13,14,15,18,26,34] find their use for various computer vision tasks such as texture-synthesis [14,15], image super-resolution [18], action recognition [13], or scene completion [24]. Scene Composites: Our work draws similarity to idea of scene composites [31,43,48,47]. Russell et al [47] use shapes or scene composites to query matches for seman-tic segmentation using LabelMe dataset [49].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Shapes and Context: In-The-Wild Image Synthesis & Manipulation

Bansal

Sheikh

Ramanan

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

input output input output input output input outputs input outputs Figure 1. Our approach synthesize images from label maps by non-parametric matching of shapes, parts, and pixels. We show example results for diverse "in-the-wild" scenes containing large amounts of variation in object composition and deformation. We also show multiple outputs generated from a label map using our approach in the bottom row. AbstractWe introduce a data-driven approach for interactively synthesizing in-the-wild images from semantic label maps. Our approach is dramatically different from recent work in this space, in that we make use of no learning. Instead, our approach uses simple but classic tools for matching scene context, shapes, and parts to a stored library of exemplars. Though simple, this approach has several notable advantages over recent work: (1) because nothing is learned, it is not limited to specific training data distributions (such as cityscapes, facades, or faces); (2) it can synthesize arbitrarily high-resolution images, limited only by the resolution of the exemplar library; (3) by appropriately composing shapes and parts, it can generate an exponentially large set of viable candidate output images (that can say, be interactively searched by a user). We present results on the diverse COCO dataset, significantly outperforming learningbased approaches on standard image synthesis metrics. Finally, we explore user-interaction and user-controllability, demonstrating that our system can be used as a platform for user-driven content creation.

show abstract

“…In contrast, our goal is to achieve the same level of photo-realism from a single RGBD input. To tackle this, we resort to a semi-parametric approach [40], where a calibration phase is used to acquire frames of the users appearance from a few different viewpoints. These calibration images are then merged together with the the current view of the user in an end-to-end fashion.…”

Section: Related Workmentioning

confidence: 99%

Volumetric Capture of Humans With a Single RGBD Camera via Semi-Parametric Learning

Pandey

Keskin

Izadi

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Volumetric (4D) performance capture is fundamental for AR/VR content generation. Whereas previous work in 4D performance capture has shown impressive results in studio settings, the technology is still far from being accessible to a typical consumer who, at best, might own a single RGBD sensor. Thus, in this work, we propose a method to synthesize free viewpoint renderings using a single RGBD camera.The key insight is to leverage previously seen "calibration" images of a given user to extrapolate what should be rendered in a novel viewpoint from the data available in the sensor. Given these past observations from multiple viewpoints, and the current RGBD image from a fixed view, we propose an end-to-end framework that fuses both these data sources to generate novel renderings of the performer. We demonstrate that the method can produce high fidelity images, and handle extreme changes in subject pose and camera viewpoints. We also show that the system generalizes to performers not seen in the training data. We run exhaustive experiments demonstrating the effectiveness of the proposed semi-parametric model (i.e. calibration images available to the neural network) compared to other state of the art machine learned solutions. Further, we compare the method with more traditional pipelines that employ multi-view capture. We show that our framework is able to achieve compelling results, with substantially less infrastructure than previously required.

show abstract

Semi-Parametric Image Synthesis

Cited by 184 publications

References 32 publications

Mask-Guided Portrait Editing With Conditional GANs

Mask-Guided Portrait Editing With Conditional GANs

Shapes and Context: In-The-Wild Image Synthesis & Manipulation

Volumetric Capture of Humans With a Single RGBD Camera via Semi-Parametric Learning

Contact Info

Product

Resources

About