Palette: Image-to-Image Diffusion Models

Saharia, Chitwan; Chan, William; Chang, Huiwen; Lee, Chris A.; Ho, Jonathan; Salimans, Tim; Fleet, David J.; Norouzi, Mohammad

doi:10.1145/3528233.3530757

Cited by 1,000 publications

(487 citation statements)

References 41 publications

Supporting

Mentioning

483

Contrasting

Unclassified

Order By: Relevance

“…Diffusion-based generative models have emerged as a powerful new framework for neural image synthesis, in both unconditional [15,33,42] and conditional [16,32,33,35,36,37,38,42] settings, even surpassing the quality of GANs [12] in certain situations [9]. They are also rapidly finding use in other domains such as audio [27,34] and video [18] generation, image segmentation [4,50] and language translation [31].…”

Section: Introductionmentioning

confidence: 99%

Elucidating the Design Space of Diffusion-Based Generative Models

Karras¹,

Aittala²,

Aila³

et al. 2022

Preprint

View full text Add to dashboard Cite

We argue that the theory and practice of diffusion-based generative models are currently unnecessarily convoluted and seek to remedy the situation by presenting a design space that clearly separates the concrete design choices. This lets us identify several changes to both the sampling and training processes, as well as preconditioning of the score networks. Together, our improvements yield new state-of-the-art FID of 1.79 for CIFAR-10 in a class-conditional setting and 1.97 in an unconditional setting, with much faster sampling (35 network evaluations per image) than prior designs. To further demonstrate their modular nature, we show that our design changes dramatically improve both the efficiency and quality obtainable with pre-trained score networks from previous work, including improving the FID of an existing ImageNet-64 model from 2.07 to near-SOTA 1.55.Preprint. Under review.

show abstract

Section: Introductionmentioning

confidence: 99%

Elucidating the Design Space of Diffusion-Based Generative Models

Karras¹,

Aittala²,

Aila³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We find that L 1 norm gives slightly better FID scores than L 2 . However, subsequent work shows that L 2 tends to generate greater diversity in SR3 outputs [34].…”

Section: Cascaded High-resolution Image Synthesismentioning

confidence: 97%

“…We expect both of these variants to work reasonably well if p(γ) is modified to account for the scale of the loss function. Further investigation of the loss function for training the denoising model is an interesting area for future research (e.g., see [34]).…”

Section: Optimizing the Denoising Modelmentioning

confidence: 99%

Image Super-Resolution Via Iterative Refinement

Saharia

Chan

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

711

219

View full text Add to dashboard Cite

We present SR3, an approach to image Super-Resolution via Repeated Refinement. SR3 adapts denoising diffusion probabilistic models [1], [2] to image-to-image translation, and performs super-resolution through a stochastic iterative denoising process. Output images are initialized with pure Gaussian noise and iteratively refined using a U-Net architecture that is trained on denoising at various noise levels, conditioned on a low-resolution input image. SR3 exhibits strong performance on super-resolution tasks at different magnification factors, on faces and natural images. We conduct human evaluation on a standard 8× face super-resolution task on CelebA-HQ for which SR3 achieves a fool rate close to 50%, suggesting photo-realistic outputs, while GAN baselines do not exceed a fool rate of 34%. We evaluate SR3 on a 4× super-resolution task on ImageNet, where SR3 outperforms baselines in human evaluation and classification accuracy of a ResNet-50 classifier trained on high-resolution images. We further show the effectiveness of SR3 in cascaded image generation, where a generative model is chained with super-resolution models to synthesize high-resolution images with competitive FID scores on the class-conditional 256×256 ImageNet generation challenge.

show abstract

“…The denoising process can be extended for the purpose of conditional generation by adding other signals as the condition [10]. When implementing the network backbone as a UNet, diffusion models are well suited for image-like data and achieved state-of-the-art results in image generation [44,46,12], superresolution [49,25,35], image-to-image translation [48] and so on. In this paper, Fig.…”

Section: Related Workmentioning

confidence: 99%

DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Cameras

Shao¹,

Zheng²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose DiffuStereo, a novel system using only sparse cameras (8 in this work) for high-quality 3D human reconstruction. At its core is a novel diffusion-based stereo module, which introduces diffusion models, a type of powerful generative models, into the iterative stereo matching network. To this end, we design a new diffusion kernel and additional stereo constraints to facilitate stereo matching and depth estimation in the network. We further present a multi-level stereo network architecture to handle high-resolution (up to 4k) inputs without requiring unaffordable memory footprint. Given a set of sparse-view color images of a human, the proposed multi-level diffusion-based stereo network can produce highly accurate depth maps, which are then converted into a high-quality 3D human model through an efficient multi-view fusion strategy. Overall, our method enables automatic reconstruction of human models with quality on par to high-end dense-view camera rigs, and this is achieved using a much more light-weight hardware setup. Experiments show that our method outperforms state-of-the-art methods by a large margin both qualitatively and quantitatively.

show abstract

Palette: Image-to-Image Diffusion Models

Cited by 1,000 publications

References 41 publications

Elucidating the Design Space of Diffusion-Based Generative Models

Elucidating the Design Space of Diffusion-Based Generative Models

Image Super-Resolution Via Iterative Refinement

DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Cameras

Contact Info

Product

Resources

About