Semantic Image Manipulation Using Scene Graphs

Dhamo, Helisa; Farshad, Azade; Laina, Iro; Navab, Nassir; Hager, Gregory D.; Tombari, Federico; Rupprecht, Christian

doi:10.1109/cvpr42600.2020.00526

Cited by 91 publications

(37 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On top, attributes highlight the properties of the object in more detail but are rarely used in practice. The effectiveness of scene graphs has been demonstrated when solving different scene understanding tasks including image retrieval (Liu et al 2007;Johnson et al 2015), scene captioning (Yang et al 2019), visual question answering (Teney et al 2017) or image generation from graphs alone (Johnson et al 2018), interactively (Ashual and Wolf 2019) or for image editing tasks (Mittal et al 2019;Dhamo et al 2020). Many of these methods either rely, or build upon, image-based scene graph prediction, a particularly well studied problem (Lu et al 2016a;Peyre et al 2017;Xu et al 2017;Newell and Deng 2017;Li et al 2017;Yang et al 2018;Zellers et al 2018;Li et al 2018c;Herzig et al 2018;Zareian et al 2020).…”

Section: Related Workmentioning

confidence: 99%

“…support relations (Nathan Silberman Derek Hoiem and Fergus 2012). Such a representation is frequently used in the image domain for higher-level task such as partial (Wang et al 2014) and full image retrieval (Johnson et al 2015), image generation (Johnson et al 2018) or even manipulation (Mittal et al 2019;Dhamo et al 2020). While 2D scene graph datasets such as Visual Genome (Krishna et al 2017) or NYUv2 (Nathan Silberman Derek Hoiem and Fergus 2012) are widely available and feature relationships between scene instances and often instance attributes, scene graphs in 3D have not been explored much.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning 3D Semantic Scene Graphs with Instance Embeddings

2022

Self Cite

View full text Add to dashboard Cite

A 3D scene is more than the geometry and classes of the objects it comprises. An essential aspect beyond object-level perception is the scene context, described as a dense semantic network of interconnected nodes. Scene graphs have become a common representation to encode the semantic richness of images, where nodes in the graph are object entities connected by edges, so-called relationships. Such graphs have been shown to be useful in achieving state-of-the-art performance in image captioning, visual question answering and image generation or editing. While scene graph prediction methods so far focused on images, we propose instead a novel neural network architecture for 3D data, where the aim is to learn to regress semantic graphs from a given 3D scene. With this work, we go beyond object-level perception, by exploring relations between object entities. Our method learns instance embeddings alongside a scene segmentation and is able to predict semantics for object nodes and edges. We leverage 3DSSG, a large scale dataset based on 3RScan that features scene graphs of changing 3D scenes. Finally, we show the effectiveness of graphs as an intermediate representation on a retrieval task.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Learning 3D Semantic Scene Graphs with Instance Embeddings

2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Johnson et al [18] introduced the reverse task of image generation from scene graphs, using a 2D layout as an intermediate representation between graphs and images, where layouts are decoded to images using a Cascade Refinement Network (CRN) [4] architecture. Later, a similar architecture was explored for image generation in an interactive form [1] as well as for semantic image manipulation [6]. Herzig et al [12] proposed a model that uses canonical scene graphs, to improve robustness in terms of graph size and noise.…”

Section: Related Workmentioning

confidence: 99%

MIGS: Meta Image Generation from Scene Graphs

Farshad¹,

Musatian²,

Dhamo³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Different from the aforementioned methods, Dhamo et al [6] focused on image manipulation and produced modified images from the edited scene graphs. Yet, this method requires both the original image and the corresponding scene graph as the supervision, it cannot generate target samples freely.…”

Section: 3mentioning

confidence: 99%

Global-Affine and Local-Specific Generative Adversarial Network for semantic-guided image generation

Zhang¹,

Ni²,

Hou³

et al. 2021

MFC

View full text Add to dashboard Cite

The recent progress in learning image feature representations has opened the way for tasks such as label-to-image or text-to-image synthesis. However, one particular challenge widely observed in existing methods is the difficulty of synthesizing fine-grained textures and small-scale instances. In this paper, we propose a novel Global-Affine and Local-Specific Generative Adversarial Network (GALS-GAN) to explicitly construct global semantic layouts and learn distinct instance-level features. To achieve this, we adopt the graph convolutional network to calculate the instance locations and spatial relationships from scene graphs, which allows our model to obtain the highfidelity semantic layouts. Also, a local-specific generator, where we introduce the feature filtering mechanism to separately learn semantic maps for different categories, is utilized to disentangle and generate specific visual features. Moreover, we especially apply a weight map predictor to better combine the global and local pathways considering the highly complementary between these two generation sub-networks. Extensive experiments on the COCO-Stuff and Visual Genome datasets demonstrate the superior generation performance of our model against previous methods, our approach is more capable of capturing photo-realistic local characteristics and rendering small-sized entities with more details.

show abstract

Semantic Image Manipulation Using Scene Graphs

Cited by 91 publications

References 31 publications

Learning 3D Semantic Scene Graphs with Instance Embeddings

Learning 3D Semantic Scene Graphs with Instance Embeddings

MIGS: Meta Image Generation from Scene Graphs

Global-Affine and Local-Specific Generative Adversarial Network for semantic-guided image generation

Contact Info

Product

Resources

About