Towards Real-World Blind Face Restoration with Generative Facial Prior

Wang, Xintao; Liu, Yu; Zhang, Honglun; Shan, Ying

doi:10.1109/cvpr46437.2021.00905

Cited by 366 publications

(252 citation statements)

References 49 publications

Supporting

Mentioning

251

Contrasting

Unclassified

Order By: Relevance

“…Although these methods can generate realistic images, they all contain a time-consuming optimization process. Addressing this issue, [3,41,48] propose to learn a posterior distribution with a pretrained StyleGAN generator. Specifically, they learn an encoder to project LR images to a latent space shared with the pretrained generator that outputs HR images.…”

Section: Related Workmentioning

confidence: 99%

“…In this way, realistic and rich textures can be generated as the decoder inputs are the mapped expressive quantized HR features f tp k instead of the raw LR features f e i . Residual Spatial Feature Transformation Facilitating the generation process of decoder G t D , we propose Residual Spatial Feature Transformation (RSFT), which extends the Spatial Feature Transform (SFT) [41,43] by adding extra shortcut connections (Fig. 1) that fuse content and texture features at decoder layers.…”

Section: Super-resolution With Quantized Texture Priormentioning

confidence: 99%

“…Reconstruction Loss We follow [9,41] and employ L1 and perceptual losses as our reconstruction loss, formulated as…”

Section: Training Objectivesmentioning

confidence: 99%

“…Their performance is therefore determined by the reference HR images, which can be difficult to locate and are not always available. Addressing this issue, recent works [35,41] turned to implicit texture priors implemented by pretrained GANs. Although bypassing the needs of explicit HR references, these methods are limited to the domain of the pretrained GANs (e.g.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we propose a novel framework, namely QuanTexSR, for blind SR of natural images. The distinct advantage of our framework is that it addresses the aforementioned limitation of previous works [3,35,41,48] by incorporating quantized texture priors (QTP) that are more suitable for natural image SR. Inspired by the recent VQ-VAE [34,38] and VQGAN [9], we define our QTP as a discrete codebook consisting of a pre-defined number of feature vectors.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Real-World Blind Super-Resolution via Feature Matching with Implicit High-Resolution Priors

Chen¹,

Shi²,

Qin³

et al. 2022

Preprint

View full text Add to dashboard Cite

A key challenge of blind image super resolution is to recover realistic textures for low-resolution images with unknown degradations. Most recent works completely rely on the generative ability of GANs, which are difficult to train. Other methods resort to high-resolution image references that are usually not available. In this work, we propose a novel framework, denoted as QuanTexSR, to restore realistic textures with the Quantized Texture Priors encoded in Vector Quantized GAN. The QuanTexSR generates textures by aligning the textureless content features to the quantized feature vectors, i.e., a pretrained feature codebook. Specifically, QuanTexSR formulates the texture generation as a feature matching problem between textureless features and a pretrained feature codebook. The final textures are then generated by the quantized features from the codebook. Since features in the codebook have shown the ability to generate natural textures in the pretrain stage, QuanTexSR can generate rich and realistic textures with the pretrained codebook as texture priors. Moreover, we propose a semantic regularization technique that regularizes the pre-training of the codebook using clusters of features extracted from the pretrained VGG19 network. This further improves texture generation with semantic context. Experiments demonstrate that the proposed QuanTexSR can generate competitive or better textures than previous approaches.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Super-resolution With Quantized Texture Priormentioning

confidence: 99%

“…Reconstruction Loss We follow [9,41] and employ L1 and perceptual losses as our reconstruction loss, formulated as…”

Section: Training Objectivesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Real-World Blind Super-Resolution via Feature Matching with Implicit High-Resolution Priors

Chen¹,

Shi²,

Qin³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Wav2Lip‐HR: Synthesising clear high‐resolution talking head in the wild

Liang,

Wang,

Chen

et al. 2023

Computer Animation & Virtual

View full text Add to dashboard Cite

Talking head generation aims to synthesize a photo‐realistic speaking video with accurate lip motion. While this field has attracted more attention in recent audio‐visual researches, most existing methods do not achieve the simultaneous improvement of lip synchronization and visual quality. In this paper, we propose Wav2Lip‐HR, a neural‐based audio‐driven high‐resolution talking head generation method. With our technique, all required to generate a clear high‐resolution lip sync talking video is an image/video of the target face and an audio clip of any speech. The primary benefit of our method is that it generates clear high‐resolution videos with sufficient facial details, rather than the ones just be large‐sized with less clarity. We first analyze key factors that limit the clarity of generated videos and then put forth several important solutions to address the problem, including data augmentation, model structure improvement and a more effective loss function. Finally, we employ several efficient metrics to evaluate the clarity of images generated by our proposed approach as well as several widely used metrics to evaluate lip‐sync performance. Numerous experiments demonstrate that our method has superior performance on visual quality and lip synchronization when compared to other existing schemes.

show abstract

Automating computational design with generative AI

Ploennigs,

Berger

2024

Civil Engineering Design

View full text Add to dashboard Cite

AI image generators based on diffusion models have recently garnered attention for their capability to create images from simple text prompts. However, for practical use in civil engineering they need to be able to create specific construction plans for given constraints. This paper investigates the potential of current AI generators in addressing such challenges, specifically for the creation of simple floor plans. We explain how the underlying diffusion‐models work and propose novel refinement approaches to improve semantic encoding and generation quality. In several experiments we show that we can improve validity of generated floor plans from 6% to 90%. Based on these results we derive future research challenges considering building information modeling. With this we provide: (i) evaluation of current generative AIs; (ii) propose improved refinement approaches; (iii) evaluate them on various examples; (iv) derive future directions for diffusion models in civil engineering.

show abstract

Towards Real-World Blind Face Restoration with Generative Facial Prior

Cited by 366 publications

References 49 publications

Real-World Blind Super-Resolution via Feature Matching with Implicit High-Resolution Priors

Real-World Blind Super-Resolution via Feature Matching with Implicit High-Resolution Priors

Wav2Lip‐HR: Synthesising clear high‐resolution talking head in the wild

Automating computational design with generative AI

Contact Info

Product

Resources

About