FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling

Xie, Christopher; Park, Keunhong; Martin-Brualla, Ricardo; Brown, Matthew A.

doi:10.1109/3dv53792.2021.00104

Cited by 44 publications

(19 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…from one or more images, with controllable shape, appearance, and environmental conditions, and the quality of the sampled images varies. Other approaches learn pose, shape, and texture from images for certain categories of objects [7,21], or interpolation, view synthesis, and segmentation of sampled category instances [42]. However, none of these approaches allow for the level of structure and material decomposition suitable for high-fidelity rendering and relighting.…”

Section: Related Workmentioning

confidence: 99%

NeROIC: Neural Rendering of Objects from Online Image Collections

Kuang¹,

Olszewski²,

Chai³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present a novel method to acquire object representations from online image collections, capturing high-quality geometry and material properties of arbitrary objects from photographs with varying cameras, illumination, and backgrounds. This enables various object-centric rendering applications such as novel-view synthesis, relighting, and harmonized background composition from challenging in-thewild input. Using a multi-stage approach extending neural radiance fields, we first infer the surface geometry and refine the coarsely estimated initial camera parameters, while leveraging coarse foreground object masks to improve the training efficiency and geometry quality. We also introduce a robust normal estimation technique which eliminates the effect of geometric noise while retaining crucial details. Lastly, we extract surface material properties and ambient illumination, represented in spherical harmonics with extensions that handle transient elements, e.g. sharp shadows. The union of these components results in a highly modular and efficient object acquisition framework. Extensive evaluations and comparisons demonstrate the advantages of our approach in capturing high-quality geometry and appearance properties useful for rendering applications. Our code will be released at https://formyfamily.github. io/NeROIC/ * This work was performed while the author was an intern at Snap, Inc. Online Images NeROIC OutputsNovel View Synthesis Relighting Composition

show abstract

Section: Related Workmentioning

confidence: 99%

NeROIC: Neural Rendering of Objects from Online Image Collections

Kuang¹,

Olszewski²,

Chai³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This physics and machine learning framework has been highly effective and subsequent works have added additional physics-based priors such as reflectance models [88], [89], normal estimation [90], and shadow models [91] to enable better novel-view synthesis and 3D reconstruction. Moreover, these physics-based priors are now also used to train on classical computer vision tasks, such as object classification and segmentation, and show improved performance over purely data-driven techniques [92] [93]. Neural de-rendering has also been used for unsupervised representation learning, and has led to improved downstream accuracy over purely data-driven methods [94].…”

Section: Neural Renderingmentioning

confidence: 99%

Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging

Klinghoffer¹,

Somasundaram²,

Tiwary³

et al. 2022

Preprint

View full text Add to dashboard Cite

Cameras were originally designed using physics-based heuristics to capture aesthetic images. In recent years, there has been a transformation in camera design from being purely physics-driven to increasingly data-driven and task-specific. In this paper, we present a framework to understand the building blocks of this nascent field of end-to-end design of camera hardware and algorithms. As part of this framework, we show how methods that exploit both physics and data have become prevalent in imaging and computer vision, underscoring a key trend that will continue to dominate the future of task-specific camera design. Finally, we share current barriers to progress in end-to-end design, and hypothesize how these barriers can be overcome.

show abstract

“…As pixelNeRF leverages local image features to synthesis novel views, it is (in contrast to ours) trained on at least two views of the same instance. In the recent work FiG-NeRF [40], Xie et al introduce a 2-component, deformable neural radiance field for jointly modeling object categories and a foreground/background segmentation. In [30], Ost et al learn a scene graph to represent automotive data enabling novel views.…”

Section: Related Workmentioning

confidence: 99%

AutoRF: Learning 3D Object Radiance Fields from Single View Observations

Muller¹,

Simonelli²,

Porzi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Figure 1. Overview of AutoRF. Our model consists of an encoder that extracts a shape and an appearance code from an object's image, which can be decoded into an implicit radiance field operating in normalized object space and leveraged for novel view synthesis. Object images are generated from real-world imagery by leveraging machine-generated 3D object detections and panoptic segmentation. At test time, we fit objects to respective target instances using a photo-metric loss formulation.

show abstract

FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling

Cited by 44 publications

References 31 publications

NeROIC: Neural Rendering of Objects from Online Image Collections

NeROIC: Neural Rendering of Objects from Online Image Collections

Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging

AutoRF: Learning 3D Object Radiance Fields from Single View Observations

Contact Info

Product

Resources

About