AttrLostGAN: Attribute Controlled Image Synthesis from Reconfigurable Layout and Style

Frolov, Stanislav; Sharma, Avneesh; Hees, J.J. van; Karayil, Tushar; Raue, Federico; Dengel, Andreas

doi:10.1007/978-3-030-92659-5_23

Cited by 13 publications

(9 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent developments focused on better instance representations [38], context-awareness [9], and improving the mask prediction of overlapping and nearby objects [24]. Recently, [27,6] enabled more explicit appearance control of individual objects by conditioning on attributes. However, the set of attributes is limited and lacks the ability to model complex interactions between objects.…”

Section: Related Workmentioning

confidence: 99%

“…Our code is based on the official repositories of [36,37,6], and we use the pre-trained BERT model from [41] as our text encoder. For training, we use the Adam [17] optimizer with β 1 = 0.0 and β 2 = 0.999.…”

Section: Training Objectives and Implementation Detailsmentioning

confidence: 99%

“…We also compute the Fréchet Inception Distance (FID) [11] which measures the distance between feature distributions of real and generated images. Following [38,6], we evaluate the visual fidelity of individual image regions by applying the IS and FID on image crops, denoted as SceneIS [6] and SceneFID [38].…”

Section: Real Imagesmentioning

confidence: 99%

“…Recently, layout-to-image methods [46,37] have attracted much interest by conditioning on a spatial layout of bounding boxes and class labels to allow the user to create a complex scene of multiple objects. To gain control over the specific appearance of objects, [27,6] further improved the methods by providing additional attributes to the corresponding objects (e.g. "red bus").…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

DT2I: Dense Text-to-Image Generation from Region Descriptions

Фролов¹,

Bansal²,

Hees³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Despite astonishing progress, generating realistic images of complex scenes remains a challenging problem. Recently, layout-to-image synthesis approaches have attracted much interest by conditioning the generator on a list of bounding boxes and corresponding class labels. However, previous approaches are very restrictive because the set of labels is fixed a priori. Meanwhile, text-to-image synthesis methods have substantially improved and provide a flexible way for conditional image generation. In this work, we introduce dense text-to-image (DT2I) synthesis as a new task to pave the way toward more intuitive image generation. Furthermore, we propose DTC-GAN, a novel method to generate images from semantically rich region descriptions, and a multi-modal region feature matching loss to encourage semantic image-text matching. Our results demonstrate the capability of our approach to generate plausible images of complex scenes using region captions.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Training Objectives and Implementation Detailsmentioning

confidence: 99%

Section: Real Imagesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

DT2I: Dense Text-to-Image Generation from Region Descriptions

Фролов¹,

Bansal²,

Hees³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the image domain, both coarse-(e.g., sentence [68]) and fine-level (e.g., layout [70] and instance attribute [17]) control signals have been explored. The progress on the video side, on the other hand, has generally been more modest, in part due to an added challenge of synthesizing temporally coherent content.…”

Section: Introductionmentioning

confidence: 99%

Layered Controllable Video Generation

Huang¹,

Jin²,

Yi³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce layered controllable video generation, where we, without any supervision, decompose the initial frame of a video into foreground and background layers, with which the user can control the video generation process by simply manipulating the foreground mask. The key challenges are the unsupervised foregroundbackground separation, which is ambiguous, and ability to anticipate user manipulations with access to only raw video sequences. We address these challenges by proposing a two-stage learning procedure. In the first stage, with the rich set of losses and dynamic foreground size prior, we learn how to separate the frame into foreground and background layers and, conditioned on these layers, how to generate the next frame using VQ-VAE generator. In the second stage, we fine-tune this network to anticipate edits to the mask, by fitting (parameterized) control to the mask from future frame. We demonstrate the effectiveness of this learning and the more granular control mechanism, while illustrating state-of-the-art performance on two benchmark datasets. We provide a video abstract as well as some video results on gabrielhuang.github.io/layered controllable video generation.

show abstract