Object-Centric Image Generation from Layouts

Abstract

Despite recent impressive results on single-object and singledomain image generation, the generation of complex scenes with multiple objects remains challenging. In this paper, we start with the idea that a model must be able to understand individual objects and relationships between objects in order to generate complex scenes well. Our layoutto-image-generation method, which we call Object-Centric Generative Adversarial Network (or OC-GAN), relies on a novel Scene-Graph Similarity Module (SGSM). The SGSM learns representations of the spatial relationships between objects in the scene, which lead to our model's improved layout-fidelity. We also propose changes to the conditioning mechanism of the generator that enhance its object instance-awareness. Apart from improving image quality, our contributions mitigate two failure modes in previous approaches: (1) spurious objects being generated without corresponding bounding boxes in the layout, and (2) overlapping bounding boxes in the layout leading to merged objects in images. Extensive quantitative evaluation and ablation studies demonstrate the impact of our contributions, with our model outperforming previous state-of-theart approaches on both the COCO-Stuff and Visual Genome datasets. Finally, we address an important limitation of evaluation metrics used in previous works by introducing SceneFID -an object-centric adaptation of the popular Fréchet Inception Distance metric, that is better suited for multi-object images.

High-Resolution Complex Scene Synthesis with Transformers

“…The final images are then produced via a series of convolutional "refinement" layers [13,2] or a pre-trained SPADE [21] model. In contrast, [26,27,28] generate scenes in a GAN setting. The generator is conditioned on the scene layout and mask predictors via a layout-aware norm, inspired by StyleGAN [15].…”

“…The generator is conditioned on the scene layout and mask predictors via a layout-aware norm, inspired by StyleGAN [15]. [28] further introduce a differentiable graph-scene matching loss to enhance the separation of objects. Orthogonal to these approaches, we avoid intermediate steps and after compressing high-frequency image details (e.g.…”

“…To capture the inherent ambiguity of this task, typically generative models are used, which learn the distribution of images conditioned on the given information. A particular class of conditional models deals with synthesis of complex scenes from coarse layouts, often specified as scene graphs [13,2,11] or bounding boxes [28,27]. In addition to learning the shape and appearance of single objects, these models are also explicitly forced to model the interactions between objects.…”

“…However, achieving high-resolution synthesis results of complex scenes remains a challenging task. We hypothesize that this is partly due to the fact that most concurrent approaches rely on many steps and auxiliary losses such as scene-matching or object-specific losses [13,2,28,11], which turns out limiting the models' overall performance.…”

Panoptic-aware Image-to-Image Translation

“…Object-level image-to-image translation. The objectlevel I2I translation is derived from the development of object-driven image generation research, e.g., synthesizing images from object scenes [1,12] and synthesizing images from scene layouts [32,34,39]. The object-level I2I translation methods utilize object perception (bounding boxes or masks) as guidance, which is effective for generating sharp object boundaries.…”