Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models

Abstract

BedroomsLiving RoomsOffices Bathrooms Figure 1. Synthetic virtual scenes generated by our method. Our model can generate a large variety of such scenes, as well as complete partial scenes, in under two seconds per scene. This performance is enabled by a pipeline of multiple deep convolutional generative models which analyze a top-down representation of the scene. AbstractWe present a new, fast and flexible pipeline for indoor scene synthesis that is based on deep convolutional generative models. Our method operates on a top-down image-based representation, and inserts objects iteratively into the scene by predicting their category, location, orientation and size with separate neural network modules. Our pipeline naturally supports automatic completion of partial scenes, as well as synthesis of complete scenes. Our method is significantly faster than the previous image-based method and generates result that outperforms state-of-the-art generative scene models in terms of faithfulness to training data and perceived visual quality.

VR content creation and exploration with deep learning: A survey

“…A living-room-first strategy based on Refs. [95] and [90] was used to determine room connections and positions; it improves the plausibility of resulted floor plans. An encoder-decoder network was then applied to predict wall positions.…”

PlanIT

“…More recently, with the availability of large datasets of 3D environments such as SUNCG [Song et al 2017], learning-based approaches have become popular. A variety of approaches have been proposed using: human-centric probabilistic grammars [Qi et al 2018], Generative Adversarial Networks trained on a matrix representation of present scene objects , recursive neural networks trained to sample 3D scene hierarchies , and convolutional neural networks (CNNs) trained on top-down image representations of rooms [Ritchie et al 2019;Wang et al 2018]. Our system uses the fast CNN modules of the latter image-based method to instantiate relationship graphs, modified significantly to work with a relationship graph as input.…”

“…Using neural networks provides robustness to the noise present in relation graphs both in the training data and for the output graphs of our generative model. We adapt modules from recent work on scene synthesis [Ritchie et al 2019], modifying them to take the graph as input and to attempt to adhere to the structure that it mandates. Section 6 describes these modules, and our overall scene instantiation search procedure, in more detail.…”

“…The second family of models is space-oriented: they instead treat space as a first-class entity, modeling what occupies each point in space. Here, space typically takes the form of a regular grid, as in recent imagebased scene synthesis models [Ritchie et al 2019;Wang et al 2018]. This modeling dichotomy is analogous to the division between Lagrangian and Eulerian simulation methods.…”

“…Our system starts with a large set of unstructured 3D scenes and automatically extracts relationship graphs using geometric and statistical heuristics. To instantiate these graphs into concrete scenes, we use a neurally-guided search procedure building upon convolutional neural network (CNN)-based scene synthesis modules from recent work [Ritchie et al 2019]. To learn how to generate the graphs themselves, we again leverage convolution as an operator for capturing context.…”