KNN-Diffusion: Image Generation via Large-Scale Retrieval

Ashual, Oron; Sheynin, Shelly; Polyak, Adam; Singer, Uriel; Gafni, Oran; Nachmani, Eliya; Taigman, Yaniv

doi:10.48550/arxiv.2204.02849

Cited by 5 publications

(10 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other works like Cogview [8], and M6 [25] have also achieved very promising results based on autoregressive models. [17,27,18,7,1,37], video generation [20], and audio generation [23,31]. It was first proposed in [17], and the subsequent work [27] proposes a reparameterization to stabilize the training.…”

Section: Related Workmentioning

confidence: 99%

Improved Vector Quantized Diffusion Models

Tang¹,

Gu²,

Bao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Vector quantized diffusion (VQ-Diffusion) is a powerful generative model for textto-image synthesis, but sometimes can still generate low-quality samples or weakly correlated images with text input. We find these issues are mainly due to the flawed sampling strategy. In this paper, we propose two important techniques to further improve the sample quality of VQ-Diffusion. 1) We explore classifier-free guidance sampling for discrete denoising diffusion model and propose a more general and effective implementation of classifier-free guidance. 2) We present a high-quality inference strategy to alleviate the joint distribution issue in VQ-Diffusion. Finally, we conduct experiments on various datasets to validate their effectiveness and show that the improved VQ-Diffusion suppresses the vanilla version by large margins. We achieve an 8.44 FID score on MSCOCO, surpassing VQ-Diffusion by 5.42 FID score. When trained on ImageNet, we dramatically improve the FID score from 11.89 to 4.83, demonstrating the superiority of our proposed techniques. The code is released at https://github.com/microsoft/VQ-Diffusion.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Improved Vector Quantized Diffusion Models

Tang¹,

Gu²,

Bao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Concurrent Work Very recently, two concurrent approaches related to our work, unCLIP [42] and kNN-Diffusion [1], have been proposed. unCLIP produces very high quality text-image results by conditioning a diffusion model on the image representation of CLIP [40] and employing large-scale computation.…”

Section: Related Workmentioning

confidence: 99%

“…When building the set M (k) D (c text ) by directly using the CLIP encodings φ CLIP (c text ) of the actual textual description itself (top row), we interestingly see that our model generalizes to generating fictional descriptions and transferring attributions across object classes. However, when using φ CLIP (c text ) together with its k − 1 nearest neighbors from the database D as done in [1], the model does not generalize to these difficult conditional inputs (mid row). When omitting the text representation and only using the k CLIP image representations of the nearest neighbors, the results get even worse (bottom row).…”

Section: Conditional Image Generation Without Conditional Trainingmentioning

confidence: 99%

“…A. 1 Training Details representations for the retrieved neighbors, we use a f − 16 VQGAN pretrained on OpenImages from Rombach et al [44] 4 . To limit the dimensionality of the representation we resize the input images for each of the k = 4 nearest neighbors to 128 × 128 px, as this does not hurt the model's performance, resulting in a latent tensor of shape 8 × 8 × 256.…”

Section: Appendix a Implementation Detailsmentioning

confidence: 99%

See 1 more Smart Citation

Retrieval-Augmented Diffusion Models

Blattmann¹,

Rombach²,

Oktay³

et al. 2022

Preprint

View full text Add to dashboard Cite

Generative image synthesis with diffusion models has recently achieved excellent visual quality in several tasks such as text-based or class-conditional image synthesis. Much of this success is due to a dramatic increase in the computational capacity invested in training these models. This work presents an alternative approach: inspired by its successful application in natural language processing, we propose to complement the diffusion model with a retrieval-based approach and to introduce an explicit memory in the form of an external database. During training, our diffusion model is trained with similar visual features retrieved via CLIP and from the neighborhood of each training instance. By leveraging CLIP's joint image-text embedding space, our model achieves highly competitive performance on tasks for which it has not been explicitly trained, such as class-conditional or text-image synthesis, and can be conditioned on both text and image embeddings. Moreover, we can apply our approach to unconditional generation, where it achieves state-of-the-art performance. Our approach incurs low computational and memory overheads and is easy to implement. We discuss its relationship to concurrent work and will publish code and pretrained models soon.

show abstract

“…However, these models are very compute intensive and so far cannot be reused for tasks other than those for which they were trained. For this reason, in the present work we build on the recently introduced retrieval-augmented diffusion models (RDMs) [3,2], which also have the potential to significantly reduce the computational complexity required in training by providing a comparatively small generative model with a large image database: While the retrieval approach provides the (local) content, the model can now focus on learning the composition of scenes based on this content. In this extended abstract, we scale RDMs and show their capability to generate artistic images as those shown in Fig.…”

mentioning

confidence: 99%

Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models

Rombach¹,

Blattmann²,

Ommer³

2022

Preprint

View full text Add to dashboard Cite

Novel architectures have recently improved generative image synthesis leading to excellent visual quality in various tasks. Of particular note is the field of "AI-Art", which has seen unprecedented growth with the emergence of powerful multimodal models such as CLIP. By combining speech and image synthesis models, so-called "prompt-engineering" has become established, in which carefully selected and composed sentences are used to achieve a certain visual style in the synthesized image. In this note, we present an alternative approach based on retrievalaugmented diffusion models (RDMs). In RDMs, a set of nearest neighbors is retrieved from an external database during training for each training instance, and the diffusion model is conditioned on these informative samples. During inference (sampling), we replace the retrieval database with a more specialized database that contains, for example, only images of a particular visual style. This provides a novel way to "prompt" a general trained model after training and thereby specify a particular visual style. As shown by our experiments, this approach is superior to specifying the visual style within the text prompt. We open-source code and model weights at https://github.com/CompVis/latent-diffusion.

show abstract

KNN-Diffusion: Image Generation via Large-Scale Retrieval

Cited by 5 publications

References 31 publications

Improved Vector Quantized Diffusion Models

Improved Vector Quantized Diffusion Models

Retrieval-Augmented Diffusion Models

Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models

Contact Info

Product

Resources

About