CLIPstyler: Image Style Transfer with a Single Text Condition

ITstyler: Image-optimized Text-based Style Transfer

“…Recently, several studies have attempted to replace style images with texts that describe certain styles. By using CLIP, a pre-trained language-image embedding model, Kwon et al [12] proposed a patch-wise CLIP loss to align text-image pairs of source and target in the CLIP space. However, CLIPstyler trains a style-specific model for each target style, requiring extra time and resources.…”

“…We assume that the image emdeddings obtained from the CLIP's encoder can also be divided into style f s i and content f c i parts in the CLIP emdedding space. Therefore, CLIPstyler [12] can achieve text-based style transfer by utilizing CLIP's feature. It uses the text encoder to encode the text describing the style as f s t .…”

“…How to match the style features in different modalities of data is a challenge in text-based style transfer. CLIPstyler [12] utilizes the Contrastive Language-Image Pre-training (CLIP) [20] models and proposes a directional CLIP loss to align the CLIPspace direction between the text-image pairs of source and stylized result. However, for each text input, CLIPstyler needs to retrain the model, which is inefficient and unpractical.…”

Draw Your Art Dream: Diverse Digital Art Synthesis with Multimodal Guided Diffusion

“…Our goal is to recognize these descriptions automatically. CLIPstyler [38] proposed patchCLIP for transferring semantic texture information on text conditions. GLIDE [47] and DALL-E 2 [50] focus on open domain image synthesis.…”

“…CLIP [49] was presented to acquire visual concepts with natural language supervision and can provide the similarity scores between texts and images. Several works have used CLIP to steer generative models, such as GANs [21,38,48], toward user-defined text prompts. In this paper, we leverage a pre-trained CLIP model for text-driven and image-driven art paintings synthesis.…”