Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image

Abstract

We propose a computational framework to jointly parse a single RGB image and reconstruct a holistic 3D configuration composed by a set of CAD models using a stochastic grammar model. Specifically, we introduce a Holistic Scene Grammar (HSG) to represent the 3D scene structure, which characterizes a joint distribution over the functional and geometric space of indoor scenes. The proposed HSG captures three essential and often latent dimensions of the indoor scenes: i) latent human context, describing the affordance and the functionality of a room arrangement, ii) geometric constraints over the scene configurations, and iii) physical constraints that guarantee physically plausible parsing and reconstruction. We solve this joint parsing and reconstruction problem in an analysis-by-synthesis fashion, seeking to minimize the differences between the input image and the rendered images generated by our 3D representation, over the space of depth, surface normal, and object segmentation map. The optimal configuration, represented by a parse graph, is inferred using Markov chain Monte Carlo (MCMC), which efficiently traverses through the non-differentiable solution space, jointly optimizing object localization, 3D layout, and hidden human context. Experimental results demonstrate that the proposed algorithm improves the generalization ability and significantly outperforms prior methods on 3D layout estimation, 3D object detection, and holistic scene understanding.

Shallow2Deep: Indoor scene modeling by single image understanding

“…We obtain the mAP score at 11.49. From Huang et al [7]'s work, they achieved 12.07 on 15 main furniture and 8.06 on all their 30 categories. It shows that our approach achieves better performance in 'smaller' objects, which is in line with the qualitative analysis.…”

“…Moreover, the support relation is also a type of spatial relationship in scene grammar to enhance contextual bindings between objects. Other approaches implemented support inference to understand scenes with a scene graph [15,7]. However, these methods either require for depth information, or rely on hand-crafted priors or models.…”

“…Longer line segments (like layout lines) would contribute more and guide the orthogonal vanishing lines in alignment with room orientation (see Part I in Figure 6). However, an improper camera initialization, particularly in cluttered environments, would often cause faulty estimation of 3D room layout [7]. We include iterations to improve the camera parameters from the detected line segments and produce a refined room layout simultaneously.…”

“…3D Object Placement The accuracy of 3D object placement is tested using manually annotated 3D bounding boxes along with the evaluation benchmark provided by [48], where the mean average precision (mAP) of the 3D IoU between the predicted bounding boxes and the ground-truth is calculated. We align the reconstructed and ground-truth scenes to the same size by unifying the camera altitude, and compare our result with the state-of-the-art [7]. Different from their work, our method is designed for modeling full scenes with considering all indoor objects, while they adopted a sparsely annotated dataset SUN-RGBD for evaluation with their 30 object categories.…”

“…scene recognition [5]), those techniques are only able to represent a fragment knowledge of full scene context.With the lack of depth clues, prior studies reconstructed indoor scenes from a single image by exploiting shallow image features (e.g. line segments and HOG descriptors [6,4]) or introducing depth estimation [7,8] to search object models. Other works adopt Render-and-Match strategy to obtain CAD scenes with their renderings similar as input images [9].…”