Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models

Gal, Ran; Arar, Moab; Atzmon, Yuval; Bermano, Amit H.; Chechik, Gal; Cohen–Or, Daniel

doi:10.48550/arxiv.2302.12228

Cited by 10 publications

(16 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fine-tuning generative models for personalization Recent works have focused on customizing and personalizing text-to-image diffusion models by fine-tuning the text embedding [17], full weights [52], or cross-attention layers [34] using a few personalized images. Other works have also investigated training-free approaches for fast adaptation [12,18,30,73]. The idea of fine-tuning only the singular values of weight matrices was introduced by FS-GAN [50] in the GAN literature and further advanced by NaviGAN [14] with an unsupervised method for discovering semantic directions in this compact parameter space.…”

Section: Related Workmentioning

confidence: 99%

SVDiff: Compact Parameter Space for Diffusion Fine-Tuning

Han¹,

Li²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Diffusion models have achieved remarkable success in text-to-image generation, enabling the creation of highquality images from text prompts or other modalities. However, existing methods for customizing these models are limited by handling multiple personalized subjects and the risk of overfitting. Moreover, their large number of parameters is inefficient for model storage. In this paper, we propose a novel approach to address these limitations in existing textto-image diffusion models for personalization. Our method involves fine-tuning the singular values of the weight matrices, leading to a compact and efficient parameter space that reduces the risk of overfitting and language-drifting. We also propose a Cut-Mix-Unmix data-augmentation technique to enhance the quality of multi-subject image generation and a simple text-based image editing framework. Our proposed SVDiff method has a significantly smaller model size (1.7MB for StableDiffusion) compared to existing methods (vanilla DreamBooth 3.66GB, Custom Diffusion 73MB), making it more practical for real-world applications.

show abstract

Section: Related Workmentioning

confidence: 99%

SVDiff: Compact Parameter Space for Diffusion Fine-Tuning

Han¹,

Li²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Fine-tuning aims to adapt the weights of a pretrained generative model to new domains [72,46,71,41,78,34,47,79,30,35,24,44], downstream tasks [70,54,77], and test images [6,53,48,31,25,49]. Several recent works also explore fine-tuning text-toimage models to learn personalized or unseen concepts [33,17,55,18] given a few exemplar images. Similarly, model editing [5,69,19,68,45,38,40,39] aims to modify specific model weights based on users' instructions to incorporate new computational rules or new visual effects.…”

Section: Related Workmentioning

confidence: 99%

Ablating Concepts in Text-to-Image Diffusion Models

Kumari¹,

Zhang²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

“…Note how styles of different fonts are preserved by the semantic modification. , 2023Ruiz et al 2022], and explainability [Chefer et al 2021]. Despite being trained on raster images, their strong visual and semantic priors have also been shown to be successfully applied to other domains, such as motion [Tevet et al 2022], meshes [Michel et al 2021], point cloud [Zhang et al 2021], and vector graphics.…”

Section: Related Workmentioning

confidence: 99%

Word-As-Image for Semantic Typography

Shir¹,

Vinker²,

Hertz³

et al. 2023

Preprint

View full text Add to dashboard Cite

Fig. 1. A few examples of our word-as-image illustrations in various fonts and for different textual concept. The semantically adjusted letters are created completely automatically using our method, and can then be used for further creative design as we illustrate here.A word-as-image is a semantic typography technique where a word illustration presents a visualization of the meaning of the word, while also preserving its readability. We present a method to create word-as-image illustrations automatically. This task is highly challenging as it requires semantic understanding of the word and a creative idea of where and how to depict these semantics in a visually pleasing and legible manner. We rely on the remarkable ability of recent large pretrained language-vision models to distill textual concepts visually. We target simple, concise, black-and-white designs that convey the semantics clearly. We deliberately do not change the color or texture of the letters and do not use embellishments. Our method optimizes the outline of each letter to convey the desired concept, guided by a pretrained Stable Diffusion model. We incorporate additional loss terms to ensure the legibility of the text and the preservation of the style of the font. We show high quality and engaging results on numerous examples and compare to alternative techniques.

show abstract

Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models

Cited by 10 publications

References 46 publications

SVDiff: Compact Parameter Space for Diffusion Fine-Tuning

SVDiff: Compact Parameter Space for Diffusion Fine-Tuning

Ablating Concepts in Text-to-Image Diffusion Models

Word-As-Image for Semantic Typography

Contact Info

Product

Resources

About