UniTune: Text-Driven Image Editing by Fine Tuning a Diffusion Model on a Single Image

Abstract

Text-driven image generation methods have shown impressive results recently, allowing casual users to generate high quality images by providing textual descriptions. However, similar capabilities for editing existing images are still out of reach. Text-driven image editing methods usually need edit masks, struggle with edits that require significant visual changes and cannot easily keep specific details of the edited portion. In this paper we make the observation that image-generation models can be converted to image-editing models simply by fine-tuning them on a single image. We also show that initializing the stochastic sampler with a noised version of the base image before the sampling and interpolating relevant details from the base image after sampling further increase the quality of the edit operation. Combining these observations, we propose UniTune, a novel image editing method. UniTune gets as input an arbitrary image and a textual edit description, and carries out the edit while maintaining high fidelity to the input image. UniTune does not require additional inputs, like masks or sketches, and can perform multiple edits on the same image without retraining. We test our method using the Imagen model in a range of different use cases. We demonstrate that it is broadly applicable and can perform a surprisingly wide range of expressive editing operations, including those requiring significant visual changes that were previously impossible.

ConIS: Controllable Text-Driven Image Stylization with Semantic Intensity

“…Specific methods [14][15][16][17]32] of image editing can also be utilized for image stylization. Kwon et al [14] propose a diffusion-based unsupervised image translation (DiffuseIT) method, which employs a pre-trained vision transformer to decouple semantic and structural features.…”

“…With their outstanding ability to produce rich stylizations, many DMs-based methods [11][12][13] produce high quality results. Furthermore, text-driven image stylization is feasible through image editing techniques [14][15][16][17]. Nonetheless, these techniques necessitate textual requirements to describe the content images or other additional requirements.…”

M3VSNET: Unsupervised Multi-Metric Multi-View Stereo Network

“…Finetuning the entire denoising model allows the model to better learn specific features of images and more accurately interpret textual prompts, resulting in edits that more closely align with user intent. UniTune [172] finetunes the diffusion model on a single base image during the tuning phase, encouraging the model to produce images similar to the base image. During the sampling phase, a modified sampling process is used to balance fidelity to the base image and alignment to the editing prompt.…”

“…StyleDiffusion [179] introduces a Mapping Network that maps features of the input image to an embedding space aligned with the embedding space of textual prompts, effectively generating a prompt embedding. Cross-attention layers are Testing-Time Finetuning Approaches Denosing Model Finetuning UniTune [172], Custom-Edit [173], KV-Inversion [174] Embedding Finetuning…”