Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Poorly-Drawn Sketches

Xia, Weihao; Yang, Yujiu; Xue, Jing-Hao

doi:10.48550/arxiv.1911.00426

Cited by 7 publications

(12 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With this method, simple scribbles can be converted to photo-realistic edits by embedding into certain layers of StyleGAN. This application is helpful to existing interactive image processing tasks such as sketchto-image generation [141], [142], [143] and sketch based image retrieval [144], [145], which usually require denselabeled datasets.…”

Section: Interactive Generationmentioning

confidence: 99%

GAN Inversion: A Survey

Xia¹,

Zhang²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

GAN inversion aims to invert a given image back into the latent space of a pretrained GAN model, for the image to be faithfully reconstructed from the inverted code by the generator. As an emerging technique to bridge the real and fake image domains, GAN inversion plays an essential role in enabling the pretrained GAN models such as StyleGAN and BigGAN to be used for real image editing applications. Meanwhile, GAN inversion also provides insights on the interpretation of GAN's latent space and how the realistic images can be generated. In this paper, we provide an overview of GAN inversion with a focus on its recent algorithms and applications. We cover important techniques of GAN inversion and their applications to image restoration and image manipulation. We further elaborate on some trends and challenges for future directions. A curated list of GAN inversion methods, datasets, and other related information can be found at github.com/weihaox/awesome-gan-inversion.

show abstract

Section: Interactive Generationmentioning

confidence: 99%

GAN Inversion: A Survey

Xia¹,

Zhang²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The text-to-image synthesis task [64,65,30,62,70,54] further employs natural language descriptions as controls. The image-to-image translation task generates photo-realism images from visual controls, such as a sketch map [17,60], semantic label map [26,4,58], human pose [37] or another image for style transfer [15,27]. Moreover, image outpainting and inpainting [25,63] can be regarded as image synthesis conditioned on preservation control signals, where some image blocks of the desired image are already specified and need to be preserved in the generated image.…”

Section: Related Workmentioning

confidence: 99%

“…Conditional image synthesis aims to create an image according to the given control signals. With the increasing demand for flexible conditional image synthesis, various kinds of control signals have been introduced into this field, which can be divided into three main modalities: (i) textual controls (TC), including the class labels [1] and natural language descriptions [62,54]; (ii) visual controls (VC), such as a spatially-aligned sketch map for reference [17,60] or another image for style transfer [15,27]; (iii) preservation controls (PC), which require the synthesized image to preserve some given image blocks, e.g., image outpainting and inpainting [63,67].…”

Section: Introductionmentioning

confidence: 99%

M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers

Zhang¹,

Ma²,

Zhou³

et al. 2021

Preprint

View full text Add to dashboard Cite

Conditional image synthesis aims to create an image according to some multi-modal guidance in the forms of textual descriptions, reference images, and image blocks to preserve, as well as their combinations. In this paper, instead of investigating these control signals separately, we propose a new two-stage architecture, UFC-BERT, to unify any number of multi-modal controls. In UFC-BERT, both the diverse control signals and the synthesized image are uniformly represented as a sequence of discrete tokens to be processed by Transformer. Different from existing two-stage autoregressive approaches such as DALL-E and VQGAN, UFC-BERT adopts non-autoregressive generation (NAR) at the second stage to enhance the holistic consistency of the synthesized image, to support preserving specified image blocks, and to improve the synthesis speed. Further, we design a progressive algorithm that iteratively improves the non-autoregressively generated image, with the help of two estimators developed for evaluating the compliance with the controls and evaluating the fidelity of the synthesized image, respectively. Extensive experiments on a newly collected large-scale clothing dataset M2C-Fashion and a facial dataset Multi-Modal CelebA-HQ verify that UFC-BERT can synthesize high-fidelity images that comply with flexible multi-modal controls.Recently, two-stage image synthesis [42,48,3,13,47] has made great progress. The first stage learns a convolutional autoencoder with quantized latent representations for converting an image into a Preprint. Under review.

show abstract

“…The editing process is done by a CNN, which is trained on randomly masked face photos with sampled sketches and color strokes in an adversarial manner. Xia et al [131] proposed a two-stage network for sketch-based portrait synthesis. The stroke calibration network is responsible for converting the input poorly-drawn sketch to a more detailed and calibrated one that resembles edge maps.…”

Section: Retrieval-and-composition Based Approachesmentioning

confidence: 99%

Deep Image Synthesis from Intuitive User Input: A Review and Perspectives

Guo

Zhang

et al. 2021

Preprint

View full text Add to dashboard Cite

In many applications of computer graphics, art and design, it is desirable for a user to provide intuitive non-image input, such as text, sketch, stroke, graph or layout, and have a computer system automatically generate photo-realistic images that adhere to the input content. While classic works that allow such automatic image content generation have followed a framework of image retrieval and composition, recent advances in deep generative models such as generative adversarial networks (GANs), variational autoencoders (VAEs), and flow-based methods have enabled more powerful and versatile image generation tasks. This paper reviews recent works for image synthesis given intuitive user input, covering advances in input versatility, image generation methodology, benchmark datasets, and evaluation metrics. This motivates new perspectives on input representation and interactivity, cross pollination between major image generation paradigms, and evaluation and comparison of generation methods.

show abstract

Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Poorly-Drawn Sketches

Cited by 7 publications

References 49 publications

GAN Inversion: A Survey

GAN Inversion: A Survey

M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers

Deep Image Synthesis from Intuitive User Input: A Review and Perspectives

Contact Info

Product

Resources

About