aRTIC GAN: A Recursive Text-Image-Conditioned GAN

Abstract

Generative Adversarial Networks have recently demonstrated the capability to synthesize photo-realistic real-world images. However, they still struggle to offer high controllability of the output image, even if several constraints are provided as input. In this work, we present a Recursive Text-Image-Conditioned GAN (aRTIC GAN), a novel approach for multi-conditional image generation under concurrent spatial and text constraints. It employs few line drawings and short descriptions to provide informative yet human-friendly conditioning. The proposed scenario is based on accessible constraints with high degrees of freedom: sketches are easy to draw and add strong restrictions on the generated objects, such as their orientation or main physical characteristics. Text on its side is so common and expressive that easily enforces information otherwise impossible to provide with minimal illustrations, such as objects components color, color shades, etc. Our aRTIC GAN is suitable for the sequential generation of multiple objects due to its compact design. In fact, the algorithm exploits the previously generated image in conjunction with the sketch and the text caption, resulting in a recurrent approach. We developed three network blocks to tackle the fundamental problems of catching captions’ semantic meanings and of handling the trade-off between smoothing grid-pattern artifacts and visual detail preservation. Furthermore, a compact three-task discriminator (covering global, local and textual aspects) was developed to preserve a lightweight and robust architecture. Extensive experiments proved the validity of aRTIC GAN and show that the combined use of sketch and description allows us to avoid explicit object labeling.

A comprehensive review of generative adversarial networks: Fundamentals, applications, and challenges

“…Also, MRP-GAN (Qi, Fan, et al, 2021), SAM-GAN (Peng et al, 2021), DM-GAN (M. Zhu, Pan, et al, 2019), DAE-GAN (Ruan et al, 2021), KT-GAN (Tan et al, 2021), Bridge-GAN (M. Yuan & Peng, 2020), CF-GAN (Y. Zhang, Han, et al, 2022), DGattGAN (H. Zhang, Zhu, et al, 2021), PCCM-GAN (Qi, Sun, et al, 2021), aRTIC GAN (Alati et al, 2022), and CDRGAN (M. Wang et al, 2021) were proposed to generate natural images based on a descriptive texts that describe these images. Likewise, Y. Zhou (2021), M. Z.…”

“…This GAN model takes both image and a text that describes an object to generate a new image containing this object. Also, MRP‐GAN (Qi, Fan, et al, 2021), SAM‐GAN (Peng et al, 2021), DM‐GAN (M. Zhu, Pan, et al, 2019), DAE‐GAN (Ruan et al, 2021), KT‐GAN (Tan et al, 2021), Bridge‐GAN (M. Yuan & Peng, 2020), CF‐GAN (Y. Zhang, Han, et al, 2022), DGattGAN (H. Zhang, Zhu, et al, 2021), PCCM‐GAN (Qi, Sun, et al, 2021), aRTIC GAN (Alati et al, 2022), and CDRGAN (M. Wang et al, 2021) were proposed to generate natural images based on a descriptive texts that describe these images. Likewise, Y. Zhou (2021), M. Z. Khan et al (2021), and Y. Zhou and Shimada (2021) proposed GAN models to synthesize face images based on the text describing these faces.…”