Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields

Chen, Yuedong; Wu, Qianyi; Zheng, Chuanxia; Cham, Tat‐Jen; Cai, Jianfei

doi:10.1007/978-3-031-19781-9_42

Cited by 22 publications

(4 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A pioneering effort by PixelNeRF first integrated pixel features of source images into vanilla NeRF that only use position information. Additionally, IBRNet (Wang et al 2021), SRF (Chibane et al 2021) and MatchNeRF (Chen et al 2023b) contributed to scene reconstruction through the feature alignment of projected points from diverse perspectives. Besides, researchers have also explored novel forms of explicit 3D representation established from sparse images (Fang et al 2023), such as voxel mesh (Maturana and Scherer 2015;Sun, Sun, and Chen 2022;Huang et al 2019;Deng et al 2021), multiplane images (MPI) (Li et al 2021;Fontaine et al 2022), or layered depth images (LDI) (Tulsiani, Tucker, and Snavely 2018;Shih et al 2020).…”

Section: Related Workmentioning

confidence: 99%

ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field

Ni,

Yang,

Yang

et al. 2024

AAAI

View full text Add to dashboard Cite

Neural Radiance Fields (NeRF) have demonstrated impressive potential in synthesizing novel views from dense input, however, their effectiveness is challenged when dealing with sparse input. Existing approaches that incorporate additional depth or semantic supervision can alleviate this issue to an extent. However, the process of supervision collection is not only costly but also potentially inaccurate. In our work, we introduce a novel model: the Collaborative Neural Radiance Fields (ColNeRF) designed to work with sparse input. The collaboration in ColNeRF includes the cooperation among sparse input source images and the cooperation among the output of the NeRF. Through this, we construct a novel collaborative module that aligns information from various views and meanwhile imposes self-supervised constraints to ensure multi-view consistency in both geometry and appearance. A Collaborative Cross-View Volume Integration module (CCVI) is proposed to capture complex occlusions and implicitly infer the spatial location of objects. Moreover, we introduce self-supervision of target rays projected in multiple directions to ensure geometric and color consistency in adjacent regions. Benefiting from the collaboration at the input and output ends, ColNeRF is capable of capturing richer and more generalized scene representation, thereby facilitating higher-quality results of the novel view synthesis. Our extensive experimental results demonstrate that ColNeRF outperforms state-of-the-art sparse input generalizable NeRF methods. Furthermore, our approach exhibits superiority in fine-tuning towards adapting to new scenes, achieving competitive performance compared to per-scene optimized NeRF-based methods while significantly reducing computational costs. Our code is available at: https://github.com/eezkni/ColNeRF.

show abstract

Section: Related Workmentioning

confidence: 99%

ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field

Ni,

Yang,

Yang

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…HoloGAN [16] ICCV 2019 face, cat, car, LSUN camera pose unsupervised deep voxel Liao et al [42] CVPR 2020 multiple object data unconditional unsupervised 3D primitives BlockGAN [20] NeurIPS 2020 multiple object data camera pose unsupervised deep voxel GRAF [28] NeurIPS 2020 rendered chair, face, cat, bird camera matrix, pose unsupervised NeRF pi-GAN [2] CVPR 2021 face, car, CARLA camera position unsupervised NeRF GIRAFFE [32] CVPR 2021 chair, cat, face, car, church camera pose unsupervised compositional NeRF GOF [45] NeurIPS 2021 cat, car, face 3D location, camera pose unsupervised generative occupancy fields ShadeGAN [43] NeurIPS 2021 cat, face 3D location, camera pose unsupervised light field CAMPARI [39] 3DV 2021 cat, car, face, chair camera pose unsupervised decomposed NeRF StyleNeRF [1] ICLR 2022 face, cat, car camera pose unsupervised NeRF GRAM [49] CVPR 2022 cat, face, CARLA camera pose unsupervised EG3D [48] CVPR 2022 cat, face camera parameters unsupervised tri-plane 3D representation VolumeGAN [51] CVPR 2022 cat, car, face, bedroom, CARLA camera pose unsupervised StyleSDF [50] CVPR 2022 cat, face camera pose unsupervised SDF Pix2NeRF [71] CVPR 2022 face, CARLA, rendered image image unsupervised Sem2NeRF [57] ECCV 2022 cat, face semantic mask, camera pose semantic mask and image SURF-GAN [63] ECCV 2022 face camera pose unsupervised EpiGRAF [60] NeurIPS 2022 cat, face, variable-shape camera pose unsupervised tri-plane 3D representation IDE-3D [61] TOG 2022 face camera pose semantic mask and image tri-plane 3D representation control over the 3D parameters θ gives the edited result x . Here, θ could be human-interpretable attribute descriptions or a set of parameters from 3D models.…”

Section: D Generative Models (From Single-view Imagery)mentioning

confidence: 99%

“…We especially emphasize their efforts towards: 1) learning efficient and expressive geometry and appearance representations ( § 5.1.1); 2) developing accelerated and view-consistent rendering algorithms ( § 5.1.2); and 3) real-time and user-interactive editing ( § 5.1.3). We then present conditional 3D-aware generative models [57], [61], [71], [135], [136], [137] in § 5.2. In 2D generative models, "unconditional" methods are referred to as those merely inputting latent codes that are sampled from a prior distribution.…”

Section: D-aware Generative Modelsmentioning

confidence: 99%

“…Once trained, given an input image, Pix2NeRF disentangles its pose and content and renders novel views of the content. Sem2NeRF [57] (Mar 2022) takes as input a single-view 2D semantic mask and outputs a NeRF-based 3D representation that can be used to render photorealistic images in a 3D-aware view-consistent manner. AutoRF [136] (Apr 2022) focuses on novel view synthesis of objects without background.…”

Section: Conditional 3d Generative Modelsmentioning

confidence: 99%

See 1 more Smart Citation

A Survey on 3D-aware Image Synthesis

Xia¹,

Xue²

2022

Preprint

View full text Add to dashboard Cite

Recent years have seen remarkable progress in deep learning powered visual content creation. This includes 3D-aware generative image synthesis, which produces high-fidelity images in a 3D-consistent manner while simultaneously capturing compact surfaces of objects from pure image collections without the need for any 3D supervision, thus bridging the gap between 2D imagery and 3D reality. The 3D-aware generative models have shown that the introduction of 3D information can lead to more controllable image generation. The task of 3D-aware image synthesis has taken the field of computer vision by storm, with hundreds of papers accepted to top-tier journals and conferences in recent year (mainly the past two years), but there lacks a comprehensive survey of this remarkable and swift progress. Our survey aims to introduce new researchers to this topic, provide a useful reference for related works, and stimulate future research directions through our discussion section. Apart from the presented papers, we aim to constantly update the latest relevant papers along with corresponding implementations at https://weihaox.github.io/awesome-3D-aware-synthesis.

show abstract

Object-Compositional Neural Implicit Surfaces

Liu

Chen

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields

Cited by 22 publications

References 50 publications

ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field

ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field

A Survey on 3D-aware Image Synthesis

Object-Compositional Neural Implicit Surfaces

Contact Info

Product

Resources

About