Human-Centric Indoor Scene Synthesis Using Stochastic Grammar

Abstract

We present a human-centric method to sample and synthesize 3D room layouts and 2D images thereof, to obtain large-scale 2D/3D image data with perfect per-pixel ground truth. An attributed spatial And-Or graph (S-AOG) is proposed to represent indoor scenes. The S-AOG is a probabilistic grammar model, in which the terminal nodes are object entities. Human contexts as contextual relations are encoded by Markov Random Fields (MRF) on the terminal nodes. We learn the distributions from an indoor scene dataset and sample new layouts using Monte Carlo Markov Chain. Experiments demonstrate that our method can robustly sample a large variety of realistic room layouts based on three criteria: (i) visual realism comparing to a state-of-the-art room arrangement method, (ii) accuracy of the affordance maps with respect to groundtruth, and (ii) the functionality and naturalness of synthesized rooms evaluated by human subjects. The code is available at https://github.com/SiyuanQi/ human-centric-scene-synthesis. Scene category Scene component Object categoryObject and attribute ... Scenes Room Furniture Supported ObjectsStochastic grammar model has been used for parsing the hierarchical structures from images of indoor [20,47] and outdoor scenes [20], and images/videos involving humans [25,40]. In this paper, instead of using stochastic grammar for parsing, we forward sample from a grammar model to generate large variations of indoor scenes.

Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models

“…It takes on average less than 2 seconds for our model to generate a complete scene on a NVIDIA Geforce GTX 1080Ti GPU, which is two orders of magnitudes faster than the previous image based method (Deep Priors). While slower than endto-end methods such as [15], our model can also perform Method Accuracy Deep Priors [12] 84.69 Human-Centric [22] 76.18 Ours 58.75 Perturbed (1%) 50.00 Perturbed (5%) 54.69 Perturbed (10%) 64.38 Table 2. Real vs. synthetic classification accuracy for scenes generated by different methods.…”

LayoutVAE: Stochastic Scene Layout Generation From a Label Set

“…Wang et al [28] use a convolutional network that iteratively generates a 3D room scene by adding one object at a time. Qi et al [23] propose a spatial And-Or graph to represent indoor scenes, from which new scenes can be sampled. Different from most other works, they use human affordances and activity information with respect to objects in the scene to model probable spatial layouts.…”

Hierarchy Denoising Recursive Autoencoders for 3D Scene Layout Prediction

“…Hierarchical representations have been used to learn grammars in natural language and images [50], 2D scenes [46], 3D shapes [26,60], and 3D scenes [32]. A related line of work parses RGB or RGB-D scenes hierarchically using And-Or graphs [20,21,34,40,64] for a variety of tasks. For full 3D scenes, there has been very limited amount of available training data with ground truth hierarchy annotations.…”