Decomposing NeRF for Editing via Feature Field Distillation

Kobayashi, Shunsuke; Matsumoto, Emiko; Sitzmann, Vincent

doi:10.48550/arxiv.2205.15585

Cited by 9 publications

(17 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compositional 3D style transfer. Thanks to its precise geometry reconstruction, StyleRF can be seamlessly integrated with NeRF-based object segmentation [12,26,65] for compositional 3D style transfer. As shown in Fig.…”

Section: Applicationsmentioning

confidence: 99%

StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields

Liu¹,

Zhan²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

3D style transfer aims to render stylized novel views of a 3D scene with multi-view consistency. However, most existing work suffers from a three-way dilemma over accurate geometry reconstruction, high-quality stylization, and being generalizable to arbitrary new styles. We propose StyleRF (Style Radiance Fields), an innovative 3D style transfer technique that resolves the three-way dilemma by performing style transformation within the feature space of a radiance field. StyleRF employs an explicit grid of high-level features to represent 3D scenes, with which highfidelity geometry can be reliably restored via volume rendering. In addition, it transforms the grid features according to the reference style which directly leads to highquality zero-shot style transfer. StyleRF consists of two innovative designs. The first is sampling-invariant content transformation that makes the transformation invari-

show abstract

Section: Applicationsmentioning

confidence: 99%

StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields

Liu¹,

Zhan²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…N3F (Tschernezki et al, 2022) minimizes the distance between NeRF's rendered feature and 2D feature for scene editing. Most recently, (Kobayashi et al, 2022) proposes to distill the visual feature from supervised CLIP-LSeg or self-supervised DINO into a 3D feature field via an element-wise feature distance loss function. It can discover the object using a query text prompt or a patch.…”

Section: Related Workmentioning

confidence: 99%

NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes

Zhang¹,

Wang²,

Gong³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural volumetric representations have shown the potential that Multi-layer Perceptrons (MLPs) can be optimized with multi-view calibrated images to represent scene geometry and appearance without explicit 3D supervision. Object segmentation can enrich many downstream applications based on the learned radiance field. However, introducing hand-crafted segmentation to define regions of interest in a complex real-world scene is non-trivial and expensive as it acquires per view annotation. This paper carries out the exploration of self-supervised learning for object segmentation using NeRF for complex real-world scenes. Our framework, called NeRF with Self-supervised Object Segmentation (NeRF-SOS), couples object segmentation and neural radiance field to segment objects in any view within a scene. By proposing a novel collaborative contrastive loss in both appearance and geometry levels, NeRF-SOS encourages NeRF models to distill compact geometry-aware segmentation clusters from their density fields and the self-supervised pre-trained 2D visual features. The self-supervised object segmentation framework can be applied to various NeRF models that both lead to photo-realistic rendering results and convincing segmentation maps for both indoor and outdoor scenarios. Extensive results on the LLFF, BlendedMVS, CO3Dv2, and Tank & Temples datasets validate the effectiveness of NeRF-SOS. It consistently surpasses other 2D-based self-supervised baselines and predicts finer object masks than existing supervised counterparts. Please refer to the video on our project page for more details: https://zhiwenfan.github.io/NeRF-SOS/.Recently, neural volumetric rendering techniques have shown great power in scene reconstruction. Especially, neural radiance field (NeRF) and its variants (Mildenhall et al., 2020a; Barron et al., 2021) adopt multi-layer perceptrons (MLPs) to learn continuous representation and utilize calibrated multi-view images to render unseen views with fine-grained details. Besides rendering quality, the ability of scene understanding has been explored by several recent works (Vora et al., 2021;Yang et al., 2021;Zhi et al., 2021). Nevertheless, they either require dense view annotations to train a heavy 3D backbone for capturing semantic representations (Vora et al., 2021;Yang et al., 2021), or necessitate human intervention to provide sparse semantic labels (Zhi et al., 2021). Recent self-supervised object discovery approaches on neural radiance fields (Yu et al., 2021c;Stelzner et al., 2021) try to decompose objects from givens scenes on the synthetic indoor data. However, still remains a gap to be applied in complex real-world scenarios.

show abstract

“…In SURF-GAN [54], they discover controllable attributes using NeRFs for training a 3D-controllable GAN. Kobayashi et al [53] enable editing via semantic scene decomposition. While the above works tackle various image editing tasks, we focus on a different task -image blending, which requires both alignment and harmonization.…”

Section: Related Workmentioning

confidence: 99%

College students’ perceptions of AI-based writing learning tools: With a focus on Google Translate, Naver Papago, and Grammarly

Kim¹,

Han²

2021

meeso

View full text Add to dashboard Cite

Image blending aims to combine multiple images seamlessly. It remains challenging for existing 2D-based methods, especially when input images are misaligned due to differences in 3D camera poses and object shapes. To tackle these issues, we propose a 3D-aware blending method using generative Neural Radiance Fields (NeRF), including two key components: 3D-aware alignment and 3D-aware blending. For 3D-aware alignment, we first estimate the camera pose of the reference image with respect to generative NeRFs and then perform 3D local alignment for each part. To further leverage 3D information of the generative NeRF, we propose 3D-aware blending that directly blends images on the NeRF's latent representation space, rather than raw pixel space. Collectively, our method outperforms existing 2D baselines, as validated by extensive quantitative and qualitative evaluations with FFHQ and AFHQ-Cat.

show abstract

Decomposing NeRF for Editing via Feature Field Distillation

Cited by 9 publications

References 56 publications

StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields

StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields

NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes

College students’ perceptions of AI-based writing learning tools: With a focus on Google Translate, Naver Papago, and Grammarly

Contact Info

Product

Resources

About