TRGAN: Text to Image Generation Through Optimizing Initial Image

Zhao, Liang; Li, Xinwei; Huang, Pingda; Chen, Zhikui; Dai, Yanqi; Li, Tianyu

doi:10.1007/978-3-030-92307-5_76

Cited by 4 publications

(5 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DDPO has therefore been used to learn how to generate images that are more compressed or uncompressed, by minimizing or maximizing JPEG compression; more aesthetically pleasing, by maximizing LAION score (Schuhmann, 2022); or more prompt-aligned, by maximizing the similarity between the embeddings of prompt and generated image description. Improving the aesthetics of the image while preserving the text-image alignment has also been done at the prompt level (Hao et al, 2023). A language model that given human input provides an optimized prompt can be trained with PPO to maximize both an aesthetic score (from an aesthetic predictor) and a relevance score (as CLIP embedding similarity) of the image generated from the given prompt.…”

Section: Overviewmentioning

confidence: 99%

Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges

Franceschelli,

Musolesi

2024

jair

View full text Add to dashboard Cite

Generative Artificial Intelligence (AI) is one of the most exciting developments in Computer Science of the last decade. At the same time, Reinforcement Learning (RL) has emerged as a very successful paradigm for a variety of machine learning tasks. In this survey, we discuss the state of the art, opportunities and open research questions in applying RL to generative AI. In particular, we will discuss three types of applications, namely, RL as an alternative way for generation without specified objectives; as a way for generating outputs while concurrently maximizing an objective function; and, finally, as a way of embedding desired characteristics, which cannot be easily captured by means of an objective function, into the generative process. We conclude the survey with an in-depth discussion of the opportunities and challenges in this fascinating emerging area.

show abstract

Section: Overviewmentioning

confidence: 99%

Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges

Franceschelli,

Musolesi

2024

jair

View full text Add to dashboard Cite

show abstract

“…While manual prompt engineering can lead to significant progress, the process of designing prompts requires time and experience, and may not always yield optimal results. Therefore, various methods have focused on automatically searching for prompts through mining (Jiang et al 2020), parsing (Haviv, Berant, and Globerson 2021), text generation (Hao et al 2023) and LLMs (Zhu et al 2023;Chakrabarty et al 2023). Additionally, many previous works have focused on gradient-based prompt learning methods, such as (Zhou et al 2022;Wen et al 2023).…”

Section: Prompt Engineeringmentioning

confidence: 99%

“…Our work is closely related to prior research in the field of prompt engineering. For example, Google's recent study (Hao et al 2023) introduced a reinforcement learningbased approach to prompt training. However, their strategy is essentially a training methodology that can be applied to other models.…”

Section: Prompt Engineeringmentioning

confidence: 99%

“…This issue is particularly pronounced in text-to-image models, as the capacity of their text encoders is relatively limited, such as CLIP text encoder (Radford et al 2021) in Stable Diffusion. Empirical observations have also shown that common user inputs are often insufficient to produce aesthetically pleasing images using current models (Hao et al 2023).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis

Hei,

Guo,

Wang

et al. 2024

AAAI

View full text Add to dashboard Cite

Well-designed prompts have demonstrated the potential to guide text-to-image models in generating amazing images. Although existing prompt engineering methods can provide high-level guidance, it is challenging for novice users to achieve the desired results by manually entering prompts due to a discrepancy between novice-user-input prompts and the model-preferred prompts. To bridge the distribution gap between user input behavior and model training datasets, we first construct a novel Coarse-Fine Granularity Prompts dataset (CFP) and propose a novel User-Friendly Fine-Grained Text Generation framework (UF-FGTG) for automated prompt optimization. For CFP, we construct a novel dataset for text-to-image tasks that combines coarse and fine-grained prompts to facilitate the development of automated prompt generation methods. For UF-FGTG, we propose a novel framework that automatically translates user-input prompts into model-preferred prompts. Specifically, we propose a prompt refiner that continually rewrites prompts to empower users to select results that align with their unique needs. Meanwhile, we integrate image-related loss functions from the text-to-image model into the training process of text generation to generate model-preferred prompts. Additionally, we propose an adaptive feature extraction module to ensure diversity in the generated results. Experiments demonstrate that our approach is capable of generating more visually appealing and diverse images than previous state-of-the-art methods, achieving an average improvement of 5% across six quality and aesthetic metrics. Data and code are available at https://github.com/Naylenv/UF-FGTG.

show abstract

“…Previous studies (Peng et al, 2023b Optimizing Prompts. Previous studies have explored various methods to optimize prompts, such as tuning soft prompts (Qin and Eisner, 2021;Liu et al, 2023a) or training auxiliary models (Hao et al, 2022;Zhang et al, 2023b). To address the need for extensive model training, the gradientfree prompting technique CoT (Kojima et al, 2022;Zhang et al, 2022; has been proposed to enhance reasoning abilities.…”

Section: Related Workmentioning

confidence: 99%

Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation

Yang,

Li,

Liu

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Large Language Models (LLMs) have showcased impressive performance. However, due to their inability to capture relationships among samples, these frozen LLMs inevitably keep repeating similar mistakes. In this work, we propose our Tuning-free Rule Accumulation (TRAN) framework, which guides LLMs in improving their performance by learning from previous mistakes. Considering data arrives sequentially, LLMs gradually accumulate rules from incorrect cases, forming a rule collection. These rules are then utilized by the LLMs to avoid making similar mistakes when processing subsequent inputs. Moreover, the rules remain independent of the primary prompts, seamlessly complementing prompt design strategies. Experimentally, we show that TRAN improves over recent baselines by a large margin.

show abstract

TRGAN: Text to Image Generation Through Optimizing Initial Image

Cited by 4 publications

References 10 publications

Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges

Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges

A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis

Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation

Contact Info

Product

Resources

About