Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects

Yang, Bangbang; Zhang, Yinda; Li, Yijin; Cui, Zhaopeng; Fanello, Sean; Bao, Hujun; Zhang, Guofeng

doi:10.48550/arxiv.2205.02714

Cited by 1 publication

(1 citation statement)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…BlockGAN and GIRAFFE [69,70] build unconditional generative models for compositions of 3D-structured representations, but only tackle generation, not reconstruction. Some methods rely on annotations such as bounding boxes, object classes, 3D object models, or instance segmentation to recover object-centric neural radiance fields [71][72][73][74]. Several scene reconstruction methods [65][66][67]75] use direct supervision to train an object representation and detector to infer an editable 3D scene from a single frame observation.…”

Section: Related Workmentioning

confidence: 99%

Seeing 3D Objects in a Single Image via Self-Supervised Static-Dynamic Disentanglement

Sharma¹,

Tewari²,

Du³

et al. 2022

Preprint

View full text Add to dashboard Cite

Human perception reliably identifies movable and immovable parts of 3D scenes, and completes the 3D structure of objects and background from incomplete observations. We learn this skill not via labeled examples, but simply by observing objects move. In this work, we propose an approach that observes unlabeled multiview videos at training time and learns to map a single image observation of a complex scene, such as a street with cars, to a 3D neural scene representation that is disentangled into movable and immovable parts while plausibly completing its 3D structure. We separately parameterize movable and immovable scene parts via 2D neural ground plans. These ground plans are 2D grids of features aligned with the ground plane that can be locally decoded into 3D neural radiance fields. Our model is trained self-supervised via neural rendering. We demonstrate that the structure inherent to our disentangled 3D representation enables a variety of downstream tasks in street-scale 3D scenes using simple heuristics, such as extraction of objectcentric 3D representations, novel view synthesis, instance segmentation, and 3D bounding box prediction, highlighting its value as a backbone for data-efficient 3D scene understanding models. This disentanglement further enables scene editing via object manipulation such as deletion, insertion, and rigid-body motion.

show abstract