In-Place Scene Labelling and Understanding with Implicit Scene Representation

Zhi, Shuaifeng; Laidlow, Tristan; Leutenegger, Stefan; Davison, Andrew J.

doi:10.1109/iccv48922.2021.01554

Cited by 262 publications

(97 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our solution to the first problem is to add a parsing branch. Since the portrait parts of the same semantic category share similar motion patterns and texture information, it will be beneficial for the appearance and geometry learning in NeRF, which is also proven in recent implicit representation studies [73][74][75]81]. As shown in Fig.…”

Section: Semantic-aware Dynamic Ray Samplingmentioning

confidence: 93%

“…Implicit Representation Methods. Recent works leverage implicit functions for learning scene representations [26,28,34,37,55,79], where multi-layer perceptron (MLP) weights are used to represent the mapping from spatial coordinates to a signal in continuous space like occupancy [32,44,50,54], signed distance function [20,65,75], color and volume density [2,16,29,34], semantic label [24,81] and neural feature map [10,36]. A recent popular work named Neural Radiance Fields (NeRF) [34] optimizes an underlying continuous volumetric scene mapping from 5D coordinate of spatial location and view direction to implicit fields of color and density for photo-realistic view results.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

Liu¹,

Xu²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Animating high-fidelity video portrait with speech audio is crucial for virtual reality and digital entertainment. While most previous studies rely on accurate explicit structural information, recent works explore the implicit scene representation of Neural Radiance Fields (NeRF) for realistic generation. In order to capture the inconsistent motions as well as the semantic difference between human head and torso, some work models them via two individual sets of NeRF, leading to unnatural results. In this work, we propose Semantic-aware Speaking Portrait NeRF (SSP-NeRF), which creates delicate audio-driven portraits using one unified set of NeRF. The proposed model can handle the detailed local facial semantics and the global head-torso relationship through two semantic-aware modules. Specifically, we first propose a Semantic-Aware Dynamic Ray Sampling module with an additional parsing branch that facilitates audio-driven volume rendering. Moreover, to enable portrait rendering in one unified neural radiance field, a Torso Deformation module is designed to stabilize the large-scale non-rigid torso motions. Extensive evaluations demonstrate that our proposed approach renders more realistic video portraits compared to previous methods. Project page: https://alvinliu0.github.io/projects/SSP-NeRF.

show abstract

Section: Semantic-aware Dynamic Ray Samplingmentioning

confidence: 93%

Section: Related Workmentioning

confidence: 99%

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

Liu¹,

Xu²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Neural radiance fields. Recently, a variety of methods based on NeRF [94] have become popular for novel view synthesis [5,39,53,63,77,79,80,85,99,107,115,124,136,141], 3D reconstruction [8,8,15,27,35,54,60,61,102,116,132,139,142,143], generative modeling [70,90,100,121] and semantic segmentation [149]. The majority of these models demonstrate impressive results on novel view synthesis but are only applicable in the single-scene setting.…”

Section: Related Workmentioning

confidence: 99%

“…Supervision. To supervise the training of parameters τ , we employ volumetric rendering as in NeRF [94] but adapt it to render semantic maps as in Semantic-NeRF [149]:…”

Section: Semantic Reasoningmentioning

confidence: 99%

“…However, the vast majority of methods built on this approach target computer graphics applications such as novel view synthesis [80,114,136,142,146] without an interpretable semantic understanding of the scene. One notable exception is Semantic-NeRF [149], which regresses a per-3D point semantic class in addition to radiance and density. Similar to NeRF [94], this method is only applicable to novel views within the same scene and does not provide the form of generalization one expects in classical semantic segmentation: the ability to infer semantics on novel scenes.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes

Vora¹,

Radwan²,

Greff³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present NeSF, a method for producing 3D semantic fields from posed RGB images alone. In place of classical 3D representations, our method builds on recent work in implicit neural scene representations wherein 3D structure is captured by point-wise functions. We leverage this methodology to recover 3D density fields upon which we then train a 3D semantic segmentation model supervised by posed 2D semantic maps. Despite being trained on 2D signals alone, our method is able to generate 3D-consistent semantic maps from novel camera poses and can be queried at arbitrary 3D points. Notably, NeSF is compatible with any method producing a density field, and its accuracy improves as the quality of the density field improves. Our empirical analysis demonstrates comparable quality to competitive 2D and 3D semantic segmentation baselines on complex, realisticallyrendered synthetic scenes. Our method is the first to offer truly dense 3D scene segmentations requiring only 2D supervision for training, and does not require any semantic input for inference on novel scenes. We encourage the readers to visit the project website.* Denotes equal contribution.

show abstract

Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields

Chen

Zheng

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In-Place Scene Labelling and Understanding with Implicit Scene Representation

Cited by 262 publications

References 22 publications

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes

Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields

Contact Info

Product

Resources

About