GLAZE: Protecting Artists from Style Mimicry by Text-to-Image Models

Abstract

Recent text-to-image diffusion models such as MidJourney and Stable Diffusion threaten to displace many in the professional artist community. In particular, models can learn to mimic the artistic style of specific artists after "fine-tuning" on samples of their art. In this paper, we describe the design, implementation and evaluation of Glaze, a tool that enables artists to apply "style cloaks" to their art before sharing online. These cloaks apply barely perceptible perturbations to images, and when used as training data, mislead generative models that try to mimic a specific artist. In coordination with the professional artist community, we deploy user studies to more than 1000 artists, assessing their views of AI art, as well as the efficacy of our tool, its usability and tolerability of perturbations, and robustness across different scenarios and against adaptive countermeasures. Both surveyed artists and empirical CLIP-based scores show that even at low perturbation levels (p=0.05), Glaze is highly successful at disrupting mimicry under normal conditions (>92%) and against adaptive countermeasures (>85%).

Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models

“…Finally, the ability to learn artistic styles may be misused for copyright infringement. However, recent work [Shan et al 2023] has shown that it is possible to protect artwork from being copied by text-to-image generators, and we hope that future research in this direction could serve to mitigate such risks of infringement.…”

IntentTuner: An Interactive Framework for Integrating Human Intentions in Fine-tuning Text-to-Image Generative Models

“…Evaluating the quality of text-to-image generation has been a challenging and ongoing research problem due to the subjective nature of image evaluation and the inherent gap between text and image modalities [49]. Specifically for evaluating the generation after fine-tuning, there are two aspects to consider: the model's ability to replicate target concepts and its controllability in modifying concepts using different textual prompts [18].…”

“…In this regard, both preventive and reactive measures need to be developed. For example, one preventive method [49] is to add toxic noises to the original images, designed to be visually negligible yet sufficient to mislead generative models with a substantial divergence in the semantic interpretation of the content. Regarding reactive measures, researchers are expected to develop more robust models to dissect the distribution differences between generative and real data, enabling discrimination and effective data governance [8,14].…”

Ablating Concepts in Text-to-Image Diffusion Models

“…Current methods can synthesize high-quality images with remarkable generalization ability, capable of composing different instances, styles, and concepts in unseen contexts. However, as these models are often trained on copyright images, it learns to mimic various artist styles [64,61] and other copyrighted content [10].…”