Deep Generative Modeling for Scene Synthesis via Hybrid Representations

Zhang, Zaiwei; Yang, Zhan-Ying; Ma, Chongyang; Luo, Linjie; Huth, Alexander G.; Vouga, Etienne; Huang, Qixing

doi:10.1145/3381866

Cited by 69 publications

(39 citation statements)

References 73 publications

(88 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thirdly, image-based feature representation can be applied to encode different aspects of enhancement styles so that convolutional neural networks can be well adopted. For example, [3,33] use top-view rendered images to encode the category or layout information for the furniture objects in a scene. We can extend their idea to encode information for scene styles in different channels of an image.…”

Section: Discussion and Future Workmentioning

confidence: 99%

Stylistic scene enhancement GAN: mixed stylistic enhancement generation for 3D indoor scenes

et al. 2019

View full text Add to dashboard Cite

In this paper, we present stylistic scene enhancement GAN, SSE-GAN, a conditional Wasserstein GAN-based approach to automatic generation of mixed stylistic enhancements for 3D indoor scenes. An enhancement indicates factors that can influence the style of an indoor scene such as furniture colors and occurrence of small objects. To facilitate network training, we propose a novel enhancement feature encoding method, which represents an enhancement by a multi-one-hot vector, and effectively accommodates different enhancement factors. A Gumbel-Softmax module is introduced in the generator network to enable the generation of high fidelity enhancement features that can better confuse the discriminator. Experiments show that our approach is superior to the other baseline methods and successfully models the relationship between the style distribution and scene enhancements. Thus, although only trained with a dataset of room images in single styles, the trained generator can generate mixed stylistic enhancements by specifying multiple styles as the condition. Our approach is the first to apply a Gumbel-Softmax module in conditional Wasserstein GANs, as well as the first to explore the application of GAN-based models in the scene enhancement field.

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

Stylistic scene enhancement GAN: mixed stylistic enhancement generation for 3D indoor scenes

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Other recent methods have adapted deep neural networks for scene synthesis. [32] uses a Generative Adversarial Network to generate scenes in an attribute-matrix form (i.e. one column per scene object).…”

Section: Related Workmentioning

confidence: 99%

Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models

Ritchie

Wang

Lin

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

126

109

View full text Add to dashboard Cite

BedroomsLiving RoomsOffices Bathrooms Figure 1. Synthetic virtual scenes generated by our method. Our model can generate a large variety of such scenes, as well as complete partial scenes, in under two seconds per scene. This performance is enabled by a pipeline of multiple deep convolutional generative models which analyze a top-down representation of the scene. AbstractWe present a new, fast and flexible pipeline for indoor scene synthesis that is based on deep convolutional generative models. Our method operates on a top-down image-based representation, and inserts objects iteratively into the scene by predicting their category, location, orientation and size with separate neural network modules. Our pipeline naturally supports automatic completion of partial scenes, as well as synthesis of complete scenes. Our method is significantly faster than the previous image-based method and generates result that outperforms state-of-the-art generative scene models in terms of faithfulness to training data and perceived visual quality.

show abstract

“…One family of approaches to this problem attempts to place objects via optimization, typically a variant of Markov Chain Monte Carlo, with respect to learned priors, hand‐crafted constraints, or some combination of the two [YYT∗11, MSL∗11, FRS∗12, QZH∗18]. Another family of approaches trains deep generative models to output scenes, using either a latent variable model such as Variational Autoencoder [LPX∗19] or Generative Adversarial Network [ZYM∗20] or using an iterative object‐by‐object approach [WSCR18, RWL19, WLW∗19]. The iterative approaches are most similar to our setting.…”

Section: Background and Related Workmentioning

confidence: 99%

Roominoes: Generating Novel 3D Floor Plans From Existing 3D Rooms

Wang

Lei

et al. 2021

Computer Graphics Forum

View full text Add to dashboard Cite

Realistic 3D indoor scene datasets have enabled significant recent progress in computer vision, scene understanding, autonomous navigation, and 3D reconstruction. But the scale, diversity, and customizability of existing datasets is limited, and it is time‐consuming and expensive to scan and annotate more. Fortunately, combinatorics is on our side: there are enough individual rooms in existing 3D scene datasets, if there was but a way to recombine them into new layouts. In this paper, we propose the task of generating novel 3D floor plans from existing 3D rooms. We identify three sub‐tasks of this problem: generation of 2D layout, retrieval of compatible 3D rooms, and deformation of 3D rooms to fit the layout. We then discuss different strategies for solving the problem, and design two representative pipelines: one uses available 2D floor plans to guide selection and deformation of 3D rooms; the other learns to retrieve a set of compatible 3D rooms and combine them into novel layouts. We design a set of metrics that evaluate the generated results with respect to each of the three subtasks and show that different methods trade off performance on these subtasks. Finally, we survey downstream tasks that benefit from generated 3D scenes and discuss strategies in selecting the methods most appropriate for the demands of these tasks.

show abstract

Deep Generative Modeling for Scene Synthesis via Hybrid Representations

Cited by 69 publications

References 73 publications

Stylistic scene enhancement GAN: mixed stylistic enhancement generation for 3D indoor scenes

Stylistic scene enhancement GAN: mixed stylistic enhancement generation for 3D indoor scenes

Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models

Roominoes: Generating Novel 3D Floor Plans From Existing 3D Rooms

Contact Info

Product

Resources

About