Retrieval-Augmented Diffusion Models

Abstract

Generative image synthesis with diffusion models has recently achieved excellent visual quality in several tasks such as text-based or class-conditional image synthesis. Much of this success is due to a dramatic increase in the computational capacity invested in training these models. This work presents an alternative approach: inspired by its successful application in natural language processing, we propose to complement the diffusion model with a retrieval-based approach and to introduce an explicit memory in the form of an external database. During training, our diffusion model is trained with similar visual features retrieved via CLIP and from the neighborhood of each training instance. By leveraging CLIP's joint image-text embedding space, our model achieves highly competitive performance on tasks for which it has not been explicitly trained, such as class-conditional or text-image synthesis, and can be conditioned on both text and image embeddings. Moreover, we can apply our approach to unconditional generation, where it achieves state-of-the-art performance. Our approach incurs low computational and memory overheads and is easy to implement. We discuss its relationship to concurrent work and will publish code and pretrained models soon.

Learning Customized Visual Models with Retrieval-Augmented Knowledge

“…In natural language processing, several works augment large language models with external data encoded with structured language and relation representations [6,23,33,39,43,53,83]. Motivated by retrieval-augmented models in NLP, several recent works leverage visual and / or textual knowledge to improve classification [44], question answering [13,47,71,77], image generation [5,14,64,90], and multi-modal tasks simultaneously [79]. RAC [44] improves long-tail classification by retrieving from a non-parametric memory consisting of pre-encoded images and text.…”

Expired

“…Aiming to understand the composition concepts in scenes, Liu et al [175] propose a compositional architecture for diffusion-based image synthesis which generates a image by composing a set of diffusion models. Observing much of the success of diffusion models is due to the dramatic increase of training cost, Blattmann et al [176] propose to complement the diffusion model with a retrievalbased approach which incurs low computational cost. Discrete Diffusion.…”

Generative Adversarial Networks for Diverse and Explainable Text-to-Image Generation

“…However, this assumption does not hold when the entity is rare, or when the desired style is greatly different from the training style. To mitigate the significant out-of-distribution performance drop, multiple works [70], [71], [72], [73] have utilized the technique of retrieval with external database as a memory. Such a technique first gained attention in NLP [74], [75], [76], [77], [78] and, more recently, in GAN-based image synthesis [79], by turning fully parametric models into semi-parametric ones.…”

“…Such a technique first gained attention in NLP [74], [75], [76], [77], [78] and, more recently, in GAN-based image synthesis [79], by turning fully parametric models into semi-parametric ones. Motivated by this, [70] has augmented diffusion models with retrieval. A retrieval-augmented diffusion model (RDM) [70] consists of a conditional DM and an image database which is interpreted as an explicit part of the model.…”

“…Motivated by this, [70] has augmented diffusion models with retrieval. A retrieval-augmented diffusion model (RDM) [70] consists of a conditional DM and an image database which is interpreted as an explicit part of the model. With the distance measured in CLIP, k-nearest neighbors are queried for each query, i.e.…”