High-Resolution Complex Scene Synthesis with Transformers

Jahn, Manuel; Rombach, Robin; Ommer, Björn

doi:10.48550/arxiv.2105.06458

Cited by 2 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We condition the generation on the layout predicted from the SGTransformer. To this purpose, we encode the layout by mapping each object into a triplet of the form o = (c y , tl y , br y ), as done by [Jahn et al, 2021]. Here, c y represents the class index, tl y and br y the bounding box coordinates on a diagonal, top left and bottom right, respectively.…”

Section: Layout To Imagementioning

confidence: 99%

Transformer-based Image Generation from Scene Graphs

Renato¹,

Palazzo²,

Spampinato³

2023

Preprint

View full text Add to dashboard Cite

Graph-structured scene descriptions can be efficiently used in generative models to control the composition of the generated image. Previous approaches are based on the combination of graph convolutional networks and adversarial methods for layout prediction and image generation, respectively. In this work, we show how employing multi-head attention to encode the graph information, as well as using a transformer-based model in the latent space for image generation can improve the quality of the sampled data, without the need to employ adversarial models with the subsequent advantage in terms of training stability. The proposed approach, specifically, is entirely based on transformer architectures both for encoding scene graphs into intermediate object layouts and for decoding these layouts into images, passing through a lower dimensional space learned by a vector-quantized variational autoencoder. Our approach shows an improved image quality with respect to state-of-the-art methods as well as a higher degree of diversity among multiple generations from the same scene graph. We evaluate our approach on three public datasets: Visual Genome, COCO, and CLEVR. We achieve an Inception Score of 13.7 and 12.8, and an FID of 52.3 and 60.3, on COCO and Visual Genome, respectively. We perform ablation studies on our contributions to assess the impact of each component. Code is available at https://github.com/perceivelab/trf-sg2im

show abstract

Section: Layout To Imagementioning

confidence: 99%

Transformer-based Image Generation from Scene Graphs

Renato¹,

Palazzo²,

Spampinato³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Our work complements GLIGEN and incorporates its grounding token design. Our work is also in line with image generation from layouts [31,25,26,10,13]. These techniques typically operate in a closed-world environment characterized by a fixed vocabulary and access to predefined layouts.…”

Section: Related Workmentioning

confidence: 77%

“…We compare with a wide range of baselines, including open-source and closed-source models. Open-sourced models include (1) GLIGEN [12]: a fine-tuned Stable Diffusion [10] Dataset. As mentioned before, our model is finetuned on COCO [14] images and captions with LVIS [8]'s instance annotations.…”

Section: Methodsmentioning

confidence: 99%

Generative Adversarial Networks for Diverse and Explainable Text-to-Image Generation

Zhang¹

View full text Add to dashboard Cite

High-Resolution Complex Scene Synthesis with Transformers

Cited by 2 publications

References 17 publications

Transformer-based Image Generation from Scene Graphs

Transformer-based Image Generation from Scene Graphs

Generative Adversarial Networks for Diverse and Explainable Text-to-Image Generation

Contact Info

Product

Resources

About