The Creativity of Text-to-Image Generation

Abstract

Humankind is entering a novel creative era in which anybody can synthesize digital information using generative artificial intelligence (AI). Text-to-image generation, in particular, has become vastly popular and millions of practitioners produce AI-generated images and AI art online. This chapter first gives an overview of the key developments that enabled a healthy co-creative online ecosystem around text-to-image generation to rapidly emerge, followed by a high-level description of key elements in this ecosystem. A particular focus is placed on prompt engineering, a creative practice that has been embraced by the AI art community. It is then argued that the emerging co-creative ecosystem constitutes an intelligent system on its own -a system that both supports human creativity, but also potentially entraps future generations and limits future development efforts in AI. The chapter discusses the potential risks and dangers of cultivating this co-creative ecosystem, such as the bias inherent in today's training data, potential quality degradation in future image generation systems due to synthetic data becoming common place, and the potential long-term effects of text-to-image generation on people's imagination, ambitions, and development.

Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?

“…In some respect, prompt engineering can be seen as a way to program a generative language model through natural language [53]. A popular way of guiding model output through prompt engineering is for text-to-image generative models [37,48]. This 'zero-shot' or 'few-shot' approach to tasks with generative language models has also achieved state-of-the-art results on several natural language tasks [8].…”

Affect-Conditioned Image Generation

“…We contend that affect conditioning is a valuable direction for future generative models, particularly in human-AI co-creative contexts. Prompt engineering is hard and time consuming, because many creators do not know exactly what they are looking for [9], and even if they did the model may not interpret their words as they intend [13]. We propose, however, that many artists and designers have a sense of the "vibe" they desire in their finished product, and affect conditioning gives a way for them to directly target that.…”

“…In creativity support applications such as art (where it may be desirable that an image evoke a particular mood or emotion [11]), or design (where diverse solutions must be explored [12]), this significantly damages the utility of existing generative models. An ethnographic study of an online community that sprung up around a popular text-to-image generator supports this: the intent-output mismatch was found to be a significant practical challenge, so much so that the time-intensive process of collaborative prompt engineering became a major focus for the community [13].…”

Prompt Stealing Attacks Against Text-to-Image Generation Models

“…Concretely, we split each prompt by commas. The first part is regarded as the subject according to [27,33], while the rest are treated as prompt modifiers. We standardize the format of some modifiers, e.g., "8 k" to "8k," "3 d" to "3d."…”

“…Rather than relying on professional artists, text-to-image generation models empower anyone to produce digital images, such as photorealistic images and commercial drawings, by entering text descriptions called prompts. According to [18,27,33], a high-quality prompt that leads to a high-quality image should consist of a subject and several prompt modifiers. The subject is a natural language description of the image; the prompt modifiers are keywords or key phrases that are related to specific elements or styles of the image.…”